Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't upgrade w/ grafana.persistence.enabled=true #188

Open
vdice opened this issue Mar 28, 2017 · 2 comments
Open

Can't upgrade w/ grafana.persistence.enabled=true #188

vdice opened this issue Mar 28, 2017 · 2 comments
Labels

Comments

@vdice
Copy link
Member

vdice commented Mar 28, 2017

On GKE, upgrading with grafana persistence enabled leads to a Failed to attach volume error:

 $ k version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.4", GitCommit:"7243c69eb523aa4377bce883e7c0dd76b84709a1", GitTreeState:"clean", BuildDate:"2017-03-07T23:53:09Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.4", GitCommit:"7243c69eb523aa4377bce883e7c0dd76b84709a1", GitTreeState:"clean", BuildDate:"2017-03-07T23:34:32Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

 $ cat values-only-grafana-persistent.yaml
monitor:
  grafana:
    persistence:
      enabled: true
  influxdb:
    persistence:
      enabled: false

 $ helm install workflow-dev/workflow --namespace deis --name deis-workflow -f values-only-grafana-persistent.yaml
...
 $ # (wait until all up and running)

 $ helm upgrade deis-workflow -f values-only-grafana-persistent.yaml workflow/workflow
...
 $ kd get po | grep grafana
deis-monitor-grafana-1674584155-qz1pq    0/1       ContainerCreating   0          40s
deis-monitor-grafana-2030706665-cg04s    1/1       Running             0          4m

$ kd describe po deis-monitor-grafana-1674584155-qz1pq
Name:		deis-monitor-grafana-1674584155-qz1pq
Namespace:	deis
Node:		gke-vrd-default-pool-7890514e-p65q/10.240.0.22
Start Time:	Tue, 28 Mar 2017 12:10:58 -0600
Labels:		app=deis-monitor-grafana
		pod-template-hash=1674584155
Status:		Pending
IP:
Controllers:	ReplicaSet/deis-monitor-grafana-1674584155
Containers:
  deis-monitor-grafana:
    Container ID:
    Image:		quay.io/deis/grafana:v2.8.0
    Image ID:
    Port:		3500/TCP
    State:		Waiting
      Reason:		ContainerCreating
    Ready:		False
    Restart Count:	0
    Volume Mounts:
      /var/lib/grafana from grafana-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-f1372 (ro)
    Environment Variables:
      INFLUXDB_URLS:		http://$(DEIS_MONITOR_INFLUXAPI_SERVICE_HOST):$(DEIS_MONITOR_INFLUXAPI_SERVICE_PORT_TRANSPORT)
      BIND_PORT:		3500
      DEFAULT_USER:		admin
      DEFAULT_USER_PASSWORD:	admin
Conditions:
  Type		Status
  Initialized 	True
  Ready 	False
  PodScheduled 	True
Volumes:
  grafana-data:
    Type:	PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:	deis-monitor-grafana
    ReadOnly:	false
  default-token-f1372:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	default-token-f1372
QoS Class:	BestEffort
Tolerations:	<none>
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----			-------------	--------	------		-------
  1m		1m		1	{default-scheduler }			Normal		Scheduled	Successfully assigned deis-monitor-grafana-1674584155-qz1pq to gke-vrd-default-pool-7890514e-p65q
  1m		14s		7	{controller-manager }			Warning		FailedMount	Failed to attach volume "pvc-5d2452ce-13e1-11e7-b75d-42010a80013f" on node "gke-vrd-default-pool-7890514e-p65q" with: googleapi: Error 400: The disk resource 'projects/deis-sandbox/zones/us-central1-b/disks/gke-vrd-c820edac-dynam-pvc-5d2452ce-13e1-11e7-b75d-42010a80013f' is already being used by 'projects/deis-sandbox/zones/us-central1-b/instances/gke-vrd-default-pool-7890514e-0xxs'
@vdice vdice added the bug label Mar 28, 2017
@kukikiloke
Copy link

Hi, I also run into the similar error when using persistence volume for grafana.

One of my pods is stuck at pending state due to error pod (deis-monitor-grafana-465911159-nkhmp) failed to fit in any node fit failure summary on nodes : NoVolumeZoneConflict (2), PodToleratesNodeTaints (1). The followings are the zones my AWS resources running on:

Master: us-east-1a
Nodes: us-east-1c, us-east-1d
Dynamic pvc Volume: us-east-1a

It seems to be related to which zones the master/nodes are hosted on as it works by running the master, nodes, volume in different zones (don't have an example at this moment).

@Cryptophobia
Copy link

This issue was moved to teamhephy/monitor#5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants