kubeprovenance

A Kubernetes Aggregated API Server to find out Provenance/Lineage information for Kuberentes Custom Resources.

What is it?

Kubernetes custom resources extend base API to manage third-party platform elements declaratively. It is important to track chronology of declarative operations performed on custom resources to understand how these operations affect underlying platform elements - e.g. for an instance of Postgres custom resource we may want to know: how many db users were created in a month, when was password changed for a db user, etc. For this, a generic approach is needed to maintain provenance information of custom resources.

kubeprovenance is a tool that helps you find Provenance information about different Kubernetes custom resources in your cluster.

Kubeprovenance is a Kubernetes aggregated API server. It uses Kubernetes audit logs for building custom resource provenance. Provenance query operators like history, diff, bisect are defined for custom resource instance tracking. Provenance information is accessible via kubectl.

Try it Out:

1. Setting Up The Environment.

Reference: https://dzone.com/articles/easy-step-by-step-local-kubernetes-source-code-cha
ssh to your VM
sudo su -
apt-get install -y gcc make socat git wget

2. Install Golang 1.10.3:
wget https://dl.google.com/go/go1.10.3.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.10.3.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
export GOROOT=/usr/local/go

Set up your Go workspace, set the GOPATH to it. This is where all your Go code should be.
mkdir $HOME/goworkspace
mkdir $HOME/goworkspace/src
mkdir $HOME/goworkspace/bin

export GOPATH=$HOME/goworkspace

3. Install etcd3.2.18: curl -L https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz -o etcd-v3.2.18-linux-amd64.tar.gz && tar xzvf etcd-v3.2.18-linux-amd64.tar.gz && /bin/cp -f etcd-v3.2.18-linux-amd64/{etcd,etcdctl} /usr/bin && rm -rf etcd-v3.2.18-linux-amd64*

4. Install Docker
Follow steps here: reference: https://docs.docker.com/install/linux/docker-ce/ubuntu/#set-up-the-repository
docker version //check if it is installed

5. Get The Kubernetes Source Code:
git clone https://github.com/kubernetes/kubernetes $GOPATH/src/k8s.io/kubernetes
cd $GOPATH/src/k8s.io/kubernetes

6. Compile and Run Kubernetes
export KUBERNETES_PROVIDER=local
root@host: $GOPATH/src/k8s.io/kubernetes# hack/local-up-cluster.sh

In a new shell, test that it is working :
root@host: $GOPATH/src/k8s.io/kubernetes# cluster/kubectl.sh cluster-info
Kubernetes master is running at http://127.0.0.1:8080 # => works!

Add $GOPATH/src/k8s.io/kubernetes/cluster to PATH:

export PATH=$PATH:$GOPATH/src/k8s.io/kubernetes/cluster

Now, commands look like kubectl.sh get pods instead of kubectl get pods...

7. Enabling Auditing:

We have to enable auditing. reference: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/
Setting up Log backend (To be added)...

If not in kubernetes directory...
cd $GOPATH/src/k8s.io/kubernetes

vi hack/local-up-cluster.sh

line 87: Change ENABLE_APISERVER_BASIC_AUDIT to true ENABLE_APISERVER_BASIC_AUDIT=${ENABLE_APISERVER_BASIC_AUDIT:-true}

line 486: add audit-policy file to audit_args:
Now you need to add an audit-arg for the audit-policy. add the following line after audit_arg+=" --audit-log-maxbackup=0"

audit_arg += " --audit-policy-file=/root/audit-policy.yaml"

The value of --audit-policy-file is where you created your audit-policy.yaml file.
There is an example-policy for a Postgres custom resource saved in this repository.

This file defines what actions and resources will generate logs.

Reference the docs if you are looking to make one:
https://kubernetes.io/docs/tasks/debug-application-cluster/audit/
For running kubeprovenance to track only a Postgres custom resource, audit-policy would look like this:
Note: Add more rules to the audit-policy to track different or more than one custom resource:

  root@provenance:~# more audit-policy.yaml
  apiVersion: audit.k8s.io/v1beta1
  kind: Policy
  omitStages:
    - "RequestReceived"
  rules:
    - level: Request
      verbs:
        - create
        - delete
        - patch
      resources:
        - group: "postgrescontroller.kubeplus"
          version: "v1"
          resources: ["postgreses"]

Note: The audit log for your custom resource will be saved where this variable is set: APISERVER_BASIC_AUDIT_LOG=/tmp/kube-apiserver-audit.log

8. Running kubeprovenance

Install dep:
curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
Move dep executable to somewhere on your $PATH
dep version -- to verify that it is installed correctly

go get github.com/cloud-ark/kubeprovenance
cd $GOPATH/src/github.com/cloud-ark/kubeprovenance
dep ensure -v

Make sure Kubernetes is running:
$ kubectl.sh cluster-info

Now to deploy this aggregated api server use these commands:

1. Build the API Server container image:
   $ ./build-provenance-artifacts.sh
2. Deploy the API Server in your cluster:
   $ ./deploy-provenance-artifacts.sh
3. Clean-up:
   $ ./delete-provenance-artifacts.sh

9. Deploy Sample Postgres Operator

Follow the steps given here

Once the kubeprovenance API server is running, you can find provenance information by using the following commands:

1. Get list of version for a Postgres custom resource instance (client25)

kubectl.sh get --raw "/apis/kubeprovenance.cloudark.io/v1/namespaces/default/postgreses/client25/versions"

2. Get Spec history for Postgres custom resource instance

kubectl.sh get --raw "/apis/kubeprovenance.cloudark.io/v1/namespaces/default/postgreses/client25/spechistory"

3. Get diff of Postgres custom resource instance between version 1 and version 5

kubectl.sh get --raw "/apis/kubeprovenance.cloudark.io/v1/namespaces/default/postgreses/client25/diff?start=1&end=5"

4. Get diff of the field databases for a Postgres custom resource instance between version 1 and version 2

kubectl.sh get --raw "/apis/kubeprovenance.cloudark.io/v1/namespaces/default/postgreses/client25/diff?start=1&end=2&field=databases"

5. Get diff of the field users for a Postgres custom resource instance between version 1 and version 3

kubectl.sh get --raw "/apis/kubeprovenance.cloudark.io/v1/namespaces/default/postgreses/client25/diff?start=1&end=3&field=users"

6. Find out in which version the user 'pallavi' was given password 'pass123'

kubectl.sh get --raw "/apis/kubeprovenance.cloudark.io/v1/namespaces/default/postgreses/client25/bisect?field1=username&value1=pallavi&field2=password&value2=pass123"

Try it on Minikube

Note: Since audit-logging is not supported on minikube yet (kubernetes/minikube#2934), I included a static, pre-generated audit-log to use to see how it works.

1. Setting up environment.
sudo su -
apt-get install -y gcc make socat git wget
2. Install Minikube
curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.28.2/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
minikube start
minikube ip -- verify that minikube is up and running
3. Install Golang 1.10.3:
wget https://dl.google.com/go/go1.10.3.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.10.3.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
export GOROOT=/usr/local/go

Set up your Go workspace, set the GOPATH to it. This is where all your Go code should be.
mkdir $HOME/goworkspace
mkdir $HOME/goworkspace/src
mkdir $HOME/goworkspace/bin

export GOPATH=$HOME/goworkspace

4. Install etcd3.2.18: curl -L https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz -o etcd-v3.2.18-linux-amd64.tar.gz && tar xzvf etcd-v3.2.18-linux-amd64.tar.gz && /bin/cp -f etcd-v3.2.18-linux-amd64/{etcd,etcdctl} /usr/bin && rm -rf etcd-v3.2.18-linux-amd64*

5. Install Docker
Follow steps here: reference: https://docs.docker.com/install/linux/docker-ce/ubuntu/#set-up-the-repository
docker version //check if it is installed

6. Install dep:
curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
Move dep executable to somewhere on your $PATH
dep version -- to verify that it is installed correctly

7. Running kubeprovenance

go get github.com/cloud-ark/kubeprovenance
cd $GOPATH/src/github.com/cloud-ark/kubeprovenance
dep ensure -v

0. Allow Minikube to use local Docker images:
   $ eval $(minikube docker-env)
1. Build the API Server container image:
   $ ./build-provenance-artifacts.sh
2. Deploy the API Server in your cluster:
   $ ./deploy-provenance-artifacts.sh
3. Clean-up:
   $ ./delete-provenance-artifacts.sh

Once the kubeprovenance API server is running, you can find provenance information by using the following commands:

1. Get list of version for a Postgres custom resource instance (client25)

kubectl get --raw "/apis/kubeprovenance.cloudark.io/v1/namespaces/default/postgreses/client25/versions"

2. Get Spec history for Postgres custom resource instance

kubectl get --raw "/apis/kubeprovenance.cloudark.io/v1/namespaces/default/postgreses/client25/spechistory"

3. Get diff of Postgres custom resource instance between version 1 and version 5

kubectl get --raw "/apis/kubeprovenance.cloudark.io/v1/namespaces/default/postgreses/client25/diff?start=1&end=5"

4. Get diff of the field databases for a Postgres custom resource instance between version 1 and version 2

kubectl get --raw "/apis/kubeprovenance.cloudark.io/v1/namespaces/default/postgreses/client25/diff?start=1&end=2&field=databases"

5. Get diff of the field users for a Postgres custom resource instance between version 1 and version 3

kubectl get --raw "/apis/kubeprovenance.cloudark.io/v1/namespaces/default/postgreses/client25/diff?start=1&end=3&field=users"

6. Find out in which version the user 'pallavi' was given password 'pass123'

kubectl get --raw "/apis/kubeprovenance.cloudark.io/v1/namespaces/default/postgreses/client25/bisect?field1=username&value1=pallavi&field2=password&value2=pass123"

Running Unit Tests:

1. go test -v ./...

Troubleshooting tips:

1. Check that the API server Pod is running:

   $ kubectl get pods -n provenance

2. Get the Pod name from output of above command and then check logs of the container. For example:

   $ kubectl logs -n provenance kube-provenance-apiserver-klzpc -c kube-provenance-apiserver

Details:

Our experience in building this API server is here.