data is lost after petset update

[fvigotti]

Hi I'm playing with your solution and everything works fine,
except when you want to update something in the petset aside from replicas count,
when you want to update something ( ie: node-affinity annotation ) the petset must be deleted and recreated , this force cluster downtime which is not good and also isn't the biggest issue,

backing storage in my cluster is hostpath (I have no other options, cinder/gluster are too slow for my cluster), ( I've used your same pv and pvc setup )
but pv and pvc cannot be bound to specific node, in this current setup also petset distinct images cannot be bound to specific nodes
so during a petset update, all containers gets deleted, and when are recreated if containers gets placed on the same previous pod you get your data back ( with datadir-$n ) if the container $n goes on another pod ( compared to the previous run ) you will get 2 directory datadir-$n and datadir-$n(of previous run) in each pod and data for the whole cluster is reinitialized in this new pvc->nodes assignment

a solution could be something like node-binding pv|pvc ( I don't think it's yet possible on kube 1.4.x ) or a way to update the petset group without deleting all pods and recreate them , also I don't think that's yet possible ..

I am missing something?

[fvigotti]

I think that untill data gravity problem is resolved on kubernetes (

• [WIP] Sticky emptydir proposal kubernetes/kubernetes#30044
• Data awareness and Data/Pod affinity kubernetes/enhancements#139
• There should be a local storage persistent volume kubernetes/kubernetes#7562
  )
  there is no solution to this using petset if you want to serve data from local disk on nodes the only rude way to have something that works correctly ( but is less cluster-friendly ) is manually create 1 rc ( with pod affinity to specific node ) and 1 svc for each kafka instance

[solsson]

I think you're right. There is no solution with hostPath, bar mechanisms to force pod or pv to a specific node. I'd try to avoid such requirements anyway, to keep nodes disposable.

Me too have been evaluating NFS and gluster, but for now I'm sticking with big cloud providers' volume types that do follow pods around.

Ok if I close this? I don't think it's a flaw with Stateful Set or the clustering in this repo.

[allenj]

I hope I'm not hijacking this conversation but the issue description seems relevant to what I'm seeing when testing your solution. I'm also using big cloud provider volume types but it looks like the way the kafka configuration is set up, none of the log data will be persisted on the volumes specified in the PetSet.

Specifically the log.dirs property. Shouldn't this point to a path on the data-dir volume in the PetSet definition?

I've modified this setting to be /opt/kafka/data/logs and built my own version of your kubernetes-persistent image so logs are saved to a data volume instead of sitting in the container space.

See: https://github.com/allenj/kubernetes-kafka/blob/0fa0d9961859d5285061a5b34054fe404815cbf9/docker-kafka-persistent/config/server.properties#L63

Let me know if this is at all accurate and if you want me to create a new issue/PR.

[fvigotti]

@allenj
I think that the this is a different problem, anyway there are some missing best practice points in this repo kafka installation ( ie : custom configuration taken from configmaps ) at the moment the configuration is the default taken from an image not even controlled from this repo... which obviously may change independently and so control must be taken from the sysadmin somehow..
also resource limits both in pods and in kafka ( instance should be fixed with a PR ) something like

        env:
        - name: "KAFKA_HEAP_OPTS"
          value: "-Xmx3G -Xms3G"
        resources:
          limits:
            cpu: 4000m
            memory: 4000Mi
          requests:
            cpu: 1000m
            memory: 3000Mi

also petset doesn't enforce anty-affinity between pods and nodes, so adding something like

          name: anti-affinity
          hostPort: 65535

on pod templates should be useful too..

@solsson
Yes you can close this :) maybe a reference about the still-open-issue with current kubernetes version 1.4 may speedup testing your solution ( which i find elegant anyway ) for those like me that are constrained on bare-metal ?

Thank you,
Francesco Vigotti

[solsson]

Thanks @allenj. I did the image change you suggested, but avoided the logs in the path because the word tends to mean something else with most services, and that something else tends to be disposable with docker :)

@fvigotti I like your suggestions. Will you crate two PRs, or should I?

I've postponed work on the PetSet kafka setup because in production we had to use https://github.com/Yolean/kubernetes-kafka/tree/nopetset-kafka due to PetSet being alpha. I will most likely revisit and re-document master as StatefulSet after the release of Kubernetes 1.5.

[fvigotti]

I would love to give you a PR but I'm struggling of too much works :( and with no time to do it , I would not be very elegant in my PR code..

If I find some spare time in next days I'll try to do something.. but anyway feel free to do by yourself or leave as it is now :) we will get a better solution anyway once data-gravity issue gets resolved by k8s :)

[juanli2]

Why I can not see the pv path on the machine?

/opt/deployment/zk-kafka-petset# zk/pv.sh
Note that in for example GKE a PetSet will have PersistentVolume(s) and PersistentVolumeClaim(s) created for it automatically
Please enter a path where to store data during local testing: (/opt/deployment/zk-kafka-petset/data)

persistentvolume "datadir-zoo-0" created
persistentvolume "datadir-zoo-1" created
persistentvolume "datadir-zoo-2" created

It claims that data path will be "/opt/deployment/zk-kafka-petset/data", and I can see this path is taking effective on the kubernetes dashboard. But why I can not see the path in my local machine>

/opt/deployment/zk-kafka-petset# cd /opt/deployment/zk-kafka-petset/data
-bash: cd: /opt/deployment/zk-kafka-petset/data: No such file or directory

[solsson]

@juanli2 Have you tried running from #12?

[allenj]

@solsson

I did the image change you suggested, but avoided the logs in the path because the word tends to mean something else with most services, and that something else tends to be disposable with docker

It might be worth mentioning that if you're using a AWS EBS for your volume and you've formatted it ext4 there will be a lost+found directory at the root level. If you mount kafka logs to be at that root the lost+found directory will confuse kafka.

[solsson]

Yes, I discovered the same with Google's persistent volumes.

[solsson]

In conclusion, with out of the box kafka config use the config override in #12. With a build of the kafka-persistent image in this repo no such override should be necessary. In both cases, mount /opt/kafka/data as a persistent volume. Big thanks to @allenj for reporting this flaw.