How to add DB certificates to a Kratos helm deploy

[ajunca]

Hi,

I'm trying to deploy Kratos in Kubernetes. I already have a distributed cockroachdb running in secure mode inside the cluster. One of the problems that I'm trying to solve now (I say one of the problems, because ory stack is fairly complicated to deploy and helm documentation is very minimal, so more issues will arise for sure) is how to deal with database certificates.

An example on cockroachdb git page example-app-secure.yaml does this in three steps:
1- An init container is executed before the actual pod and asks for csr certificates
2- Manually approving the certificate
3- On the actual pod, mount the volume that the init container ask for with the certificates and voilà

How can I reproduce this using Kratos helm chart? I tried to attach an InitContainer without success. I'm fairly lost, Kratos documentation on helm deploy is very little. As a workaround I tried to connect Kratos using an insecure database connection (using user and password already created inside the db), cockroachdb refuse the connection because is not secure, so this doesn't seems an option.

Any help would be very appreciated as I been trying this for days without success.

[ajunca]

I've been digging more and I think I just answered my self: I can use helm post-renderer (aka Kustomize) and apply some modification to the generated helm resource before it is applied to the API server. So I guess I could add an init container and mount the certificate volume afterwards to the kratos container.

I'm still confused about other init container that kratos helm has. Particularly the one is checking that the dns connection is OK(?) I'm guessing I will have to remove this check as I'm not sure how I can give the certificates to that init container...

Tomorrow I will try this but I'm still quite confused. Any more suited/easy solution will be welcomed.

[aeneasr]

@tricky42 you can probably answer this :)

[ajunca]

First of all, thank you for the fast response.

Can you share how you deployed the CockroachDB Cluster, then we can give more specific advise

I deployed cockroachdb inside a kubernetes cluster with rancher using this helm values (attached at the end of the message).

extraInitContainer(s) to be configured via values.yaml similar to e.g. the Postgres Chart.

As I understand this functionality is desired for the main project? (question linked to the next paragraph)

In order to unblock yourself and if you are familiar with Helm Chart templating you can checkout the Kratos Helm chart and change https://github.com/ory/k8s/blob/master/helm/charts/kratos/templates/deployment.yaml to support the definition of extraInitContainers basically copying the solution in the PostgreSQL chart.

To be honest I'm not really familiar with Helm (nor with kubernetes), just begin using it for this project. I can try to go this route, but just thinking that I will have to maintain a fork of the original helm with the changes it really turns me off. Do you think that If I make the appropriated changes (with your help of this community in slack) to add this functionality, then the changes can be pushed to the master branch? I ask because if not, I will probably try the post-renderer route as I don't have to maintain a fork of the chart.

Thanks again, I really appreciated the help.

Cockroachdb helm:

clusterDomain: cluster.local
conf:
  attrs: []
  cache: 25%
  cluster-name: ''
  disable-cluster-name-verification: false
  http-port: 8080
  join: []
  locality: ''
  logtostderr: INFO
  max-sql-memory: 25%
  port: 26257
  single-node: false
  sql-audit-dir: ''
image:
  credentials: {}
  pullPolicy: IfNotPresent
  repository: cockroachdb/cockroach
  tag: v20.2.5
ingress:
  annotations: {}
  enabled: false
  hosts: []
  labels: {}
  paths:
    - /
  tls: []
init:
  affinity: {}
  annotations: {}
  labels:
    app.kubernetes.io/component: init
  nodeSelector: {}
  resources: {}
  tolerations: []
labels: {}
networkPolicy:
  enabled: false
  ingress:
    grpc: []
    http: []
service:
  discovery:
    annotations: {}
    labels:
      app.kubernetes.io/component: cockroachdb
  ports:
    grpc:
      external:
        name: tcp
        port: 26257
      internal:
        name: grpc-internal
        port: 26257
    http:
      name: http
      port: 8080
  public:
    annotations: {}
    labels:
      app.kubernetes.io/component: cockroachdb
    type: ClusterIP
serviceMonitor:
  annotations: {}
  enabled: false
  interval: 10s
  labels: {}
statefulset:
  annotations: {}
  args: []
  budget:
    maxUnavailable: 1
  env: []
  labels:
    app.kubernetes.io/component: cockroachdb
  nodeAffinity: {}
  nodeSelector:
    db-storage: "true"
  podAffinity: {}
  podAntiAffinity:
    topologyKey: kubernetes.io/hostname
    type: soft
    weight: 100
  podManagementPolicy: Parallel
  priorityClassName: ''
  replicas: 3
  resources:
    requests:
      cpu: "1"
      memory: "4Gi"
    limits:
      cpu: "1"
      memory: "4Gi"
  secretMounts: []
  tolerations: []
  topologySpreadConstraints:
    maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
  updateStrategy:
    type: RollingUpdate
storage:
  hostPath: ''
  persistentVolume:
    annotations: {}
    enabled: true
    labels: 
      db-storage: "true"
    size: 100Gi
    storageClass: local-storage
tls:
  certs:
    clientRootSecret: root-secret
    nodeSecret: node-secret
    provided: false
    tlsSecret: true
  enabled: true
  init:
    image:
      credentials: {}
      pullPolicy: IfNotPresent
      repository: cockroachdb/cockroach-k8s-request-cert
      tag: '0.4'
  serviceAccount:
    create: true
    name: ''

[tricky42]

Hi @junky89,

yes it would be definitely the goal for this to be merged with the master via a PR. So think about it and let me know if you are going to give it a try then we can also connect on Slack ;)

Cheers

[ajunca]

Hi @tricky42 ,

Sorry to bother again. I did the small change to accommodate the "extraInitContainers" in the helm charts, but I'm trying to join the Slack community without success. I want to ask some questions related to this change and some other problems I'm facing trying to deploy it. I think I have to get invited or accepted somehow, but I'm not sure how. Any documentation on this?

By the way, I'm also new to git discussions, so I'm not sure If I have to mark this questions as solved/answered or someone else will. If I have to, I can't find where.

Cheers

[tricky42]

Hi @junky89,

no problem at all. In order to join our Slack Workspace you need to sign up here: https://slack.ory.sh/. You will receive an email with a link were you are asked to enter your name + set your password... The you can login to Slack and find us ;)

I am also not too familiar with Github Discussions so far, but I think there is currently no way to mark a discussion as solved.

See you soon in Slack,

Cheers

[vinckr]

I marked the discussion as solved ;)
aeneasr gets the credit though, because he opened this thread.
if you want to be credited @tricky42, you can copy your answer below and I can select that as answer then.