Operator: Type DirectoryOrCreate should be set on hostPath data volume of logs daemonset

[emadolsky]

What's wrong?

Daemonset pods of LogsInstance fail to create with this error:

Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /var/lib/grafana-agent/data

According to Kubernetes docs, if the type: DirectoryOrCreate is set on the hostPath data volume, the problem is resolved.

Steps to reproduce

Just run a logs instance for the first time.
Tested on Kubernetes v1.24.17.

System information

No response

Software version

No response

Configuration

No response

Logs

No response

[emadolsky]

This is the daemonset created by the operator:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: "8"
    meta.helm.sh/release-name: loki
    meta.helm.sh/release-namespace: centralized-logging
  creationTimestamp: "2024-05-16T09:07:14Z"
  generation: 8
  labels:
    app.kubernetes.io/instance: loki
    app.kubernetes.io/managed-by: grafana-agent-operator
    app.kubernetes.io/name: grafana-agent
    grafana-agent: loki
    operator.agent.grafana.com/name: loki
    operator.agent.grafana.com/type: logs
  name: loki-logs
  namespace: centralized-logging
  ownerReferences:
  - apiVersion: monitoring.grafana.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: GrafanaAgent
    name: loki
    uid: c6736bcc-0df5-4536-a252-25fb65ff5f6f
  resourceVersion: "282279904"
  uid: ca17c0e9-d7d6-46a8-a6b6-f06278191de4
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: loki
      app.kubernetes.io/managed-by: grafana-agent-operator
      app.kubernetes.io/name: grafana-agent
      grafana-agent: loki
      operator.agent.grafana.com/name: loki
      operator.agent.grafana.com/type: logs
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/default-container: grafana-agent
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: loki
        app.kubernetes.io/managed-by: grafana-agent-operator
        app.kubernetes.io/name: grafana-agent
        app.kubernetes.io/version: v0-39-1
        grafana-agent: loki
        operator.agent.grafana.com/name: loki
        operator.agent.grafana.com/type: logs
    spec:
      containers:
      - args:
        - --config-file=/var/lib/grafana-agent/config-in/agent.yml
        - --config-envsubst-file=/var/lib/grafana-agent/config/agent.yml
        - --watch-interval=1m
        - --statefulset-ordinal-from-envvar=POD_NAME
        - --reload-url=http://127.0.0.1:8080/-/reload
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: AGENT_DEPLOY_MODE
          value: operator
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: SHARD
          value: "0"
        image: quay.io/prometheus-operator/prometheus-config-reloader:v0.67.1
        imagePullPolicy: IfNotPresent
        name: config-reloader
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/grafana-agent/config-in
          name: config
          readOnly: true
        - mountPath: /var/lib/grafana-agent/config
          name: config-out
        - mountPath: /var/lib/grafana-agent/secrets
          name: secrets
          readOnly: true
        - mountPath: /var/log
          name: varlog
          readOnly: true
        - mountPath: /var/lib/docker/containers
          name: dockerlogs
          readOnly: true
        - mountPath: /var/lib/grafana-agent/data
          name: data
      - args:
        - -config.file=/var/lib/grafana-agent/config/agent.yml
        - -config.expand-env=true
        - -server.http.address=0.0.0.0:8080
        - -enable-features=integrations-next
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: AGENT_DEPLOY_MODE
          value: operator
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: SHARD
          value: "0"
        image: grafana/agent:v0.39.1
        imagePullPolicy: IfNotPresent
        name: grafana-agent
        ports:
        - containerPort: 8080
          name: http-metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 120
          httpGet:
            path: /-/ready
            port: http-metrics
            scheme: HTTP
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 3
        resources:
          limits:
            cpu: "1"
            memory: 2Gi
          requests:
            cpu: 500m
            memory: 1Gi
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - mountPath: /var/lib/grafana-agent/config-in
          name: config
          readOnly: true
        - mountPath: /var/lib/grafana-agent/config
          name: config-out
        - mountPath: /var/lib/grafana-agent/secrets
          name: secrets
          readOnly: true
        - mountPath: /var/log
          name: varlog
          readOnly: true
        - mountPath: /var/lib/docker/containers
          name: dockerlogs
          readOnly: true
        - mountPath: /var/lib/grafana-agent/data
          name: data
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: loki-grafana-agent
      serviceAccountName: loki-grafana-agent
      terminationGracePeriodSeconds: 4800
      volumes:
      - name: config
        secret:
          defaultMode: 420
          secretName: loki-logs-config
      - emptyDir: {}
        name: config-out
      - name: secrets
        secret:
          defaultMode: 420
          secretName: loki-secrets
      - hostPath:
          path: /var/log
          type: ""
        name: varlog
      - hostPath:
          path: /var/lib/docker/containers
          type: ""
        name: dockerlogs
      - hostPath:
          path: /var/lib/grafana-agent/data
          type: "" # This is the value that needs to be changed
        name: data
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate

[github-actions]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!