[BUG] secret "mailu-certificates" not found

[jdevries3133]

Describe the bug

Mailu frontend pod fails to startup because it cannot mount a secret named mailu-certificates

Environment

Microk8s, chart version 0.3.1

Additional context

I don't know where the mailu-certificates secret is coming from. I looked through the helm charts here and found no reference to that object in any of the manifests, so I'm a bit confused. Also, more generally, I don't see any version tags in this repository and I looked through commits a little bit and wasn't finding "version bump" commits or anything like that. I was trying to figure that out because I don't know if there is a mailu-certificates secret involved in my version of the chart, but it's since been removed. Clearly, I'm new to using helm & kubernetes, but how can I find the source for a specific version of the chart?

Also, here is the full kubectl describe output for the troublesome pod:

Name:           mailu-front-5ff698bfcb-74t89
Namespace:      mailu
Priority:       0
Node:           xxx/192.168.x.x
Start Time:     Sat, 26 Mar 2022 08:34:56 -0400
Labels:         app=mailu
                component=front
                pod-template-hash=5ff698bfcb
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/mailu-front-5ff698bfcb
Containers:
  front:
    Container ID:
    Image:          mailu/nginx:1.9.10
    Image ID:
    Ports:          110/TCP, 995/TCP, 143/TCP, 993/TCP, 25/TCP, 465/TCP, 10025/TCP, 10143/TCP, 587/TCP, 8000/TCP, 80/TCP
    Host Ports:     110/TCP, 995/TCP, 143/TCP, 993/TCP, 25/TCP, 465/TCP, 0/TCP, 0/TCP, 587/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  200Mi
    Requests:
      cpu:      100m
      memory:   100Mi
    Liveness:   http-get http://:http/health delay=0s timeout=5s period=10s #success=1 #failure=3
    Readiness:  http-get http://:http/health delay=0s timeout=5s period=10s #success=1 #failure=1
    Startup:    http-get http://:http/health delay=0s timeout=5s period=10s #success=1 #failure=30
    Environment:
      LOG_LEVEL:               WARNING
      KUBERNETES_INGRESS:      true
      TLS_FLAVOR:              cert
      HOSTNAMES:               mail.example.com,mail1.example.com
      ADMIN_ADDRESS:           mailu-admin.mailu.svc.cluster.local
      ANTISPAM_WEBUI_ADDRESS:  mailu-rspamd.mailu.svc.cluster.local:11334
      POSTMASTER:              postmaster
      DOMAIN:                  mail.example.com
      REAL_IP_HEADER:          X-Forwarded-For
      REAL_IP_FROM:            0.0.0.0/0
      WEBMAIL:                 roundcube
      WEBMAIL_ADDRESS:         mailu-roundcube.mailu.svc.cluster.local
      WEB_WEBMAIL:             /roundcube
      WEBROOT_REDIRECT:        /roundcube
      MESSAGE_SIZE_LIMIT:      52428800
      WEBDAV:                  none
      WEBDAV_ADDRESS:          localhost
      ADMIN:                   true
      WEB_ADMIN:               /admin
      SUBNET:                  10.42.0.0/16
    Mounts:
      /certs from certs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-grgs9 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  mailu-certificates
    Optional:    false
  kube-api-access-grgs9:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                 From     Message
  ----     ------       ----                ----     -------
  Warning  FailedMount  103s (x4 over 10m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[certs], unattached volumes=[certs kube-api-access-grgs9]: timed out waiting for the conditi
on
  Warning  FailedMount  46s (x15 over 15m)  kubelet  MountVolume.SetUp failed for volume "certs" : secret "mailu-certificates" not found

[jdevries3133]

Update: I realize now that the secret should be created by CertManager by this template:

helm-charts/mailu/templates/certificate.yaml

Lines 1 to 22 in 787b710

	# This is the definition of the required ssl certificate for the mail system
	# It will be issued by cert-manager
	{{ if .Values.certmanager.enabled }}
	apiVersion: {{ .Values.certmanager.apiVersion }}
	kind: Certificate
	metadata:
	name: {{ include "mailu.fullname" . }}-certificates
	spec:
	secretName: {{ include "mailu.fullname" . }}-certificates
	# re-new certificate when it expires in less than 60 days
	renewBefore: 1440h0m0s
	commonName: "{{ first (required "hostname" .Values.hostnames) }}"
	dnsNames:
	{{- range .Values.hostnames }}
	- "{{ . }}"
	{{- end }}
	issuerRef:
	kind: {{ .Values.certmanager.issuerType }}
	name: {{ .Values.certmanager.issuerName }}
	{{- end }}

I'm guessing the issue is related to CertManger being an incompatible version or using the wrong api version or something.

[Pumba98]

this https://github.com/Mailu/helm-charts/blob/master/mailu/values.yaml#L115 has to be cert-manager.io/v1 with new certmanager versions iirc
see #159

[jdevries3133]

@Pumba98 yeah, my values.yml looks like this:

hostnames:
  - mail.example.com
  - mail1.example.com

domain: mail.example.com

initialAccount:
  username: jack
  domain: mail.example.com

persistence:
  storageClass: openebs-device

roundcube:
  enabled: true

certmanager:
  enabled: true
  issuerType: ClusterIssuer
  issuerName: letsencrypt-prod
  apiVersion: cert-manager.io/v1

[jdevries3133]

I figured out the problem – a bit of nonce is being added to the end of the certificate object name, so instead of mailu-certificates, it becomes mailu-certificates-cmxls

When I look at the manifest for the secret that is created, it looks like this:

metadata:
  creationTimestamp: "2022-03-26T13:36:14Z"
  generateName: mailu-certificates-
  labels:
    cert-manager.io/next-private-key: "true"
  name: mailu-certificates-cmxls
  namespace: mailu
  ownerReferences:
  - apiVersion: cert-manager.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Certificate
    name: mailu-certificates
    uid: 6715b85e-4018-451f-8b8e-f51a3b451f3d
  resourceVersion: "10036240"
  selfLink: /api/v1/namespaces/mailu/secrets/mailu-certificates-cmxls
  uid: 008022bc-d878-4c83-8a72-d07036139865

I notice that it has a generateName field in the metadata which presumably causes this behavior. I'm not seeing that in the chart template, so I wonder if it's something unique to my CertManager setup? Not sure...

[jdevries3133]

Update: it turns out that this secret with the hash at the end of it is actually a temporary secret which is used to store information while the certificate is being issued. In my case, the certificate is not being successfully issued – I haven't figured that out yet, but it's not an issue with mailu or the helm chart. If you see something like this, inspect the CertificateRequest object, and follow that thread until figuring out why the certificate isn't being issued. I'll revise this comment when I figure out the exact problem in my case. I have a feeling it's DNS related.

[arthef]

Have you figured this out yet? It happens to me and I guess it might happen to more people. If you found solution, please add a comment for others to find.

[jdevries3133]

I don't think I ever figured it out. Later, I ran into this issue, which ultimately lead me to give up: canonical/microk8s#3025 I think that ultimately, my ingress controller wasn't exposing Mailu to the outside world, which means that the ACME challenge never completed, which is why the certificates remained in a pending state. Ultimately, if this happens to you, I think it's always *not* a problem with Mailu. It is more likely something wonky with your CertificateIssuer or Ingress Controller where the ACME challenge is not completing and the certificate is not being issued. The situation I described above are just downstream symptoms of that root issue.