Can't upgrade w/ grafana.persistence.enabled=true

[Cryptophobia]

From @vdice on March 28, 2017 18:17

On GKE, upgrading with grafana persistence enabled leads to a Failed to attach volume error:

 $ k version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.4", GitCommit:"7243c69eb523aa4377bce883e7c0dd76b84709a1", GitTreeState:"clean", BuildDate:"2017-03-07T23:53:09Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.4", GitCommit:"7243c69eb523aa4377bce883e7c0dd76b84709a1", GitTreeState:"clean", BuildDate:"2017-03-07T23:34:32Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

 $ cat values-only-grafana-persistent.yaml
monitor:
  grafana:
    persistence:
      enabled: true
  influxdb:
    persistence:
      enabled: false

 $ helm install workflow-dev/workflow --namespace deis --name deis-workflow -f values-only-grafana-persistent.yaml
...
 $ # (wait until all up and running)

 $ helm upgrade deis-workflow -f values-only-grafana-persistent.yaml workflow/workflow
...
 $ kd get po | grep grafana
deis-monitor-grafana-1674584155-qz1pq    0/1       ContainerCreating   0          40s
deis-monitor-grafana-2030706665-cg04s    1/1       Running             0          4m

$ kd describe po deis-monitor-grafana-1674584155-qz1pq
Name:		deis-monitor-grafana-1674584155-qz1pq
Namespace:	deis
Node:		gke-vrd-default-pool-7890514e-p65q/10.240.0.22
Start Time:	Tue, 28 Mar 2017 12:10:58 -0600
Labels:		app=deis-monitor-grafana
		pod-template-hash=1674584155
Status:		Pending
IP:
Controllers:	ReplicaSet/deis-monitor-grafana-1674584155
Containers:
  deis-monitor-grafana:
    Container ID:
    Image:		quay.io/deis/grafana:v2.8.0
    Image ID:
    Port:		3500/TCP
    State:		Waiting
      Reason:		ContainerCreating
    Ready:		False
    Restart Count:	0
    Volume Mounts:
      /var/lib/grafana from grafana-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-f1372 (ro)
    Environment Variables:
      INFLUXDB_URLS:		http://$(DEIS_MONITOR_INFLUXAPI_SERVICE_HOST):$(DEIS_MONITOR_INFLUXAPI_SERVICE_PORT_TRANSPORT)
      BIND_PORT:		3500
      DEFAULT_USER:		admin
      DEFAULT_USER_PASSWORD:	admin
Conditions:
  Type		Status
  Initialized 	True
  Ready 	False
  PodScheduled 	True
Volumes:
  grafana-data:
    Type:	PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:	deis-monitor-grafana
    ReadOnly:	false
  default-token-f1372:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	default-token-f1372
QoS Class:	BestEffort
Tolerations:	<none>
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----			-------------	--------	------		-------
  1m		1m		1	{default-scheduler }			Normal		Scheduled	Successfully assigned deis-monitor-grafana-1674584155-qz1pq to gke-vrd-default-pool-7890514e-p65q
  1m		14s		7	{controller-manager }			Warning		FailedMount	Failed to attach volume "pvc-5d2452ce-13e1-11e7-b75d-42010a80013f" on node "gke-vrd-default-pool-7890514e-p65q" with: googleapi: Error 400: The disk resource 'projects/deis-sandbox/zones/us-central1-b/disks/gke-vrd-c820edac-dynam-pvc-5d2452ce-13e1-11e7-b75d-42010a80013f' is already being used by 'projects/deis-sandbox/zones/us-central1-b/instances/gke-vrd-default-pool-7890514e-0xxs'

Copied from original issue: deis/monitor#188

[Cryptophobia]

From @kukikiloke on May 8, 2017 17:58

Hi, I also run into the similar error when using persistence volume for grafana.

One of my pods is stuck at pending state due to error pod (deis-monitor-grafana-465911159-nkhmp) failed to fit in any node fit failure summary on nodes : NoVolumeZoneConflict (2), PodToleratesNodeTaints (1). The followings are the zones my AWS resources running on:

Master: us-east-1a
Nodes: us-east-1c, us-east-1d
Dynamic pvc Volume: us-east-1a

It seems to be related to which zones the master/nodes are hosted on as it works by running the master, nodes, volume in different zones (don't have an example at this moment).