Terrakube on EKS with nginx-ingress #397

jderieg · 2023-06-01T22:49:30Z

jderieg
Jun 1, 2023

Hi there, I have an EKS cluster with nginx-ingress already installed. I'm trying to install Terrakube (using the helm chart), and I've used the values provided by the terrakube docs for the ingress setup. This is what mine looks like, you'll notice it's very similar to the setup in the docs:

...
## Ingress properties
ingress:
  useTls: false
  ui:
    enabled: true
    domain: "terrakube-ui.${terrakube_dnsdomain}" # Change for your real domain
    path: "/(.*)"
    pathType: "Prefix"
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/use-regex: "true"
  api:
    enabled: true
    domain: "terrakube-api.${terrakube_dnsdomain}" # Change for your real domain
    path: "/(.*)"
    pathType: "Prefix"
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: "proxy_set_header Authorization $http_authorization;"
  registry:
    enabled: true
    domain: "terrakube-reg.${terrakube_dnsdomain}" # Change for your real domain
    path: "/(.*)"
    pathType: "Prefix"
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: "proxy_set_header Authorization $http_authorization;"
  dex:
    enabled: true
    path: "/dex/(.*)"
    pathType: "Prefix"
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: "proxy_set_header Authorization $http_authorization;"

The helm chart installs fine, and the configuration gets written to my conf.d directory on the nginx-ingress pod. However, whenever I call terrakube-ui.example.com for example, it seems that nginx-ingress wants to tack on /etc/nginx/http/ to the URL path.

For example, if I were to call http://terrakube-ui.example.com/login, nginx-ingress would rewrite that URL as http://terrakube-ui.example.com/etc/nginx/http/login`, and I'm not entirely sure why. I don't see anything in the nginx config that should be doing that. Any ideas or suggestions would be appreciated.

The values file I'm using for the nginx-ingress controller helm chart are really simple. These are the only settings I'm modifying off the base install:

controller:
  replicaCount: 3
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 10
    targetCPUUtilizationPercentage: 70
    targetMemoryUtilizationPercentage: 70
  disableIPV6: true
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
      service.beta.kubernetes.io/aws-load-balancer-type: "external"

Answered by alfespa17

Jun 2, 2023

Hello @jderieg

I think this values can work for you:

## Global Name
name: "terrakube"

## Terrakube Security
security:
  useOpenLDAP: false
  adminGroup: ${terrakube_gh_group}
  patSecret: ${terrakube_dex_pat}  # Sample Key 32 characters z6QHX!y@Nep2QDT!53vgH43^PjRXyC3X
  internalSecret: ${terrakube_dex_internal} # Sample Key 32 characters Kb^8cMerPNZV6hS!9!kcD*KuUPUBa^B3
  dexClientId: "github"
  dexClientScope: "email openid profile offline_access groups"

## Terraform Storage
storage:
  aws:
    accessKey: ${awskey}
    secretKey: ${awssecret}
    bucketName: "${environment_letter_tag}-terrakube"
    region: "us-east-2"

dex:
  enabled: true
  config:
    issuer: https://terrakube-api.…

View full answer

alfespa17 · 2023-06-02T12:20:43Z

alfespa17
Jun 2, 2023
Maintainer

Hello @jderieg

What is the helm chart and terrakube version that you are using?

12 replies

jderieg Jun 2, 2023
Author

OK, I'll test out a cluster using that nginx ingress instead of the official one and see if it works.

jderieg Jun 2, 2023
Author

Thanks for the suggestion! With the kubernetes version of ingress-nginx, it seems to work. I can now get to my terraform-ui page in a browser. However, when I hit the Login button, nothing happens.
Checking the developer console in the browser, I see the following error:
"MetadataService.ts:53 Uncaught (in promise) Error: No authority or metadataUrl configured on settings"

I'm pretty sure I have all my settings correct in my values file for the helm chart. Here's my entire values file. Let me know if you see anything misconfigured or if you have any other ideas why I might be getting that error:

## Global Name
name: "terrakube"

## Terrakube Security
security:
  adminGroup: ${terrakube_gh_group} # This should be your Github team the format is OrganizationName:TeamName
  patSecret: ${terrakube_dex_pat}  # Sample Key 32 characters z6QHX!y@Nep2QDT!53vgH43^PjRXyC3X
  internalSecret: ${terrakube_dex_internal} # Sample Key 32 characters Kb^8cMerPNZV6hS!9!kcD*KuUPUBa^B3
  dexClientId: "github"
  dexClientScope: "email openid profile offline_access groups"
  dexIssuerUri: "https://terrakube-api.${terrakube_dnsdomain}/dex" # Change for your real domain

## Terraform Storage
storage:
  aws:
    accessKey: ${awskey}
    secretKey: ${awssecret}
    bucketName: "${environment_letter_tag}-terrakube"
    region: "us-east-2"

dex:
  enabled: true
  version: "v2.32.0"
  replicaCount: "1"
  serviceType: "ClusterIP"
  resources:
    limits:
      cpu: 512m
      memory: 256Mi
    requests:
      cpu: 256m
      memory: 128Mi
  properties:
    config:
      issuer: https://terrakube-api.${terrakube_dnsdomain}/dex
      storage:
        type: memory
      oauth2:
        responseTypes: ["code", "token", "id_token"]
        skipApprovalScreen: true
      web:
        allowedOrigins: ["*"]

      staticClients:
        - id: github
          redirectURIs:
            - 'https://terrakube-ui.${terrakube_dnsdomain}'
            - 'http://localhost:10001/login'
            - 'http://localhost:10000/login'
            - '/device/callback'
          name: 'github'
          public: true

      connectors:
        - type: github
          id: github
          name: gitHub
          config:
            clientID: "96bc73d25bfb9bee186d"
            clientSecret: ${terrakube_gh_clientsecret}
            redirectURI: "https://terrakube-api.${terrakube_dnsdomain}/dex/callback"
            loadAllGroups: true

## API properties
api:
  enabled: true
  version: "2.10.0"
  replicaCount: "1"
  serviceType: "ClusterIP"
  properties:
    databaseType: "H2"

## Executor properties
executor:
  enabled: true
  version: "2.10.0"
  replicaCount: "1"
  serviceType: "ClusterIP"
  properties:
    toolsRepository: "https://github.com/AzBuilder/terrakube-extensions"
    toolsBranch: "main"

## Registry properties
registry:
  enabled: true
  version: "2.10.0"
  replicaCount: "1"
  serviceType: "ClusterIP"

## UI Properties
ui:
  enabled: true
  version: "2.10.0"
  replicaCount: "1"
  serviceType: "ClusterIP"

## Ingress properties
ingress:
  useTls: true
  ui:
    enabled: true
    domain: "terrakube-ui.${terrakube_dnsdomain}" # Change for your real domain
    path: "/(.*)"
    pathType: "Prefix"
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/use-regex: "true"
      cert-manager.io/cluster-issuer: letsencrypt-${environment_tag}
  api:
    enabled: true
    domain: "terrakube-api.${terrakube_dnsdomain}" # Change for your real domain
    path: "/(.*)"
    pathType: "Prefix"
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: "proxy_set_header Authorization $http_authorization;"
      cert-manager.io/cluster-issuer: letsencrypt-${environment_tag}
  registry:
    enabled: true
    domain: "terrakube-reg.${terrakube_dnsdomain}" # Change for your real domain
    path: "/(.*)"
    pathType: "Prefix"
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: "proxy_set_header Authorization $http_authorization;"
      cert-manager.io/cluster-issuer: letsencrypt-${environment_tag}
  dex:
    enabled: true
    path: "/dex/(.*)"
    pathType: "Prefix"
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: "proxy_set_header Authorization $http_authorization;"
      cert-manager.io/cluster-issuer: letsencrypt-${environment_tag}

alfespa17 Jun 2, 2023
Maintainer

Hello @jderieg

I can see the problem, sometime ago we change how we handle the DEX configuration in the helm chart, You will have to do some small changes in the above values.yaml, let me show you some example.

Regards.

alfespa17 Jun 2, 2023
Maintainer

Hello @jderieg

I think this values can work for you:

## Global Name
name: "terrakube"

## Terrakube Security
security:
  useOpenLDAP: false
  adminGroup: ${terrakube_gh_group}
  patSecret: ${terrakube_dex_pat}  # Sample Key 32 characters z6QHX!y@Nep2QDT!53vgH43^PjRXyC3X
  internalSecret: ${terrakube_dex_internal} # Sample Key 32 characters Kb^8cMerPNZV6hS!9!kcD*KuUPUBa^B3
  dexClientId: "github"
  dexClientScope: "email openid profile offline_access groups"

## Terraform Storage
storage:
  aws:
    accessKey: ${awskey}
    secretKey: ${awssecret}
    bucketName: "${environment_letter_tag}-terrakube"
    region: "us-east-2"

dex:
  enabled: true
  config:
    issuer: https://terrakube-api.${terrakube_dnsdomain}/dex
    storage:
      type: memory
    oauth2:
      responseTypes: ["code", "token", "id_token"]
      skipApprovalScreen: true
    web:
      allowedOrigins: ["*"]

    staticClients:
    - id: github
      redirectURIs:
      - 'https://terrakube-ui.${terrakube_dnsdomain}'
      - 'http://localhost:10001/login'
      - 'http://localhost:10000/login'
      - '/device/callback'
      name: 'github'
      public: true

    connectors:
    - type: github
      id: github
      name: gitHub
      config:
        clientID: "96bc73d25bfb9bee186d"
        clientSecret: ${terrakube_gh_clientsecret}
        redirectURI: "https://terrakube-api.${terrakube_dnsdomain}/dex/callback"
        loadAllGroups: true

## Ingress properties
ingress:
  useTls: true
  ui:
    enabled: true
    domain: "terrakube-ui.${terrakube_dnsdomain}" # Change for your real domain
    path: "/(.*)"
    pathType: "Prefix"
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/use-regex: "true"
      cert-manager.io/cluster-issuer: letsencrypt-${environment_tag}
  api:
    enabled: true
    domain: "terrakube-api.${terrakube_dnsdomain}" # Change for your real domain
    path: "/(.*)"
    pathType: "Prefix"
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: "proxy_set_header Authorization $http_authorization;"
      cert-manager.io/cluster-issuer: letsencrypt-${environment_tag}
  registry:
    enabled: true
    domain: "terrakube-reg.${terrakube_dnsdomain}" # Change for your real domain
    path: "/(.*)"
    pathType: "Prefix"
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: "proxy_set_header Authorization $http_authorization;"
      cert-manager.io/cluster-issuer: letsencrypt-${environment_tag}
  dex:
    enabled: true
    path: "/dex/(.*)"
    pathType: "Prefix"
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: "proxy_set_header Authorization $http_authorization;"
      cert-manager.io/cluster-issuer: letsencrypt-${environment_tag}

I will have to update the docs, to reflect the current changes in the helm chart.

To see the latest structure you can always refer to the default values.yalm in this link.

https://github.com/AzBuilder/terrakube-helm-chart/blob/main/charts/terrakube/values.yaml

You only need to specify this section if you want to do some customization:

## API properties
api:
  enabled: true
  version: "2.10.0"
  replicaCount: "1"
  serviceType: "ClusterIP"
  properties:
    databaseType: "H2"

## Executor properties
executor:
  enabled: true
  version: "2.10.0"
  replicaCount: "1"
  serviceType: "ClusterIP"
  properties:
    toolsRepository: "https://github.com/AzBuilder/terrakube-extensions"
    toolsBranch: "main"

## Registry properties
registry:
  enabled: true
  version: "2.10.0"
  replicaCount: "1"
  serviceType: "ClusterIP"

## UI Properties
ui:
  enabled: true
  version: "2.10.0"
  replicaCount: "1"
  serviceType: "ClusterIP"

By default it will take the values from here:
https://github.com/AzBuilder/terrakube-helm-chart/blob/main/charts/terrakube/values.yaml#L157

I hope this can help you. Let me know if you face any other issue, I will be happy to help.

Answer selected by jderieg

jderieg Jun 2, 2023
Author

First off, let me just say thanks again for being so helpful and quick to respond on this! I really appreciate it.
We're getting closer. The UI is now functioning properly with the Github Auth, so that's great!
I'm using the config you sent above, and all the pods are starting properly with the exception of the API and Executor pods.
Both of those are throwing a java exception ending with the following:

Caused by: java.net.UnknownHostException: terrakube-redis-master
	at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source)
	at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
	at java.base/java.net.Socket.connect(Unknown Source)
	at redis.clients.jedis.DefaultJedisSocketFactory.createSocket(DefaultJedisSocketFactory.java:80)
	... 94 more

That seems to be caused by my custom naming of the helm install and pods. If I set that to just 'terrakube' it seems to initialize both those pods correctly.

alfespa17 Jun 2, 2023
Maintainer

Can you share the helm install command maybe I will have to do a fix so other people don't face that problem?

jderieg Jun 2, 2023
Author

I'm actually installing it with Terraform using the 'helm_release' resource. So my code looks like this:

  name             = "${var.env_tag}-${var.my_region_id}-terrakube"
  namespace        = var.namespace
  create_namespace = true
  repository       = "https://AzBuilder.github.io/terrakube-helm-chart"
  chart            = "terrakube"
  version          = var.terrakube_helmchart_version
  values = [
     ......
  ]
}

so the name in my case ends up being something like 'dev-region2-terrakube', which also names the pods similarly:
'dev-region2-terrakube-redis-master-0' for example.

For now, I'm ok just leaving it as 'terrakube', but it would be nice to be able to do a custom name like that.

alfespa17 Jun 2, 2023
Maintainer

I think the issue with the name is because of this line.

https://github.com/AzBuilder/terrakube-helm-chart/blob/main/charts/terrakube/templates/secrets-api.yaml#L19

I left the value hardcoded XD

I will fix that later

jderieg · 2023-06-02T22:04:28Z

jderieg
Jun 2, 2023
Author

I'm able to auth with Github and login now. When I go to create an organization in Terrakube, I'm getting a 403 from the API:

6 replies

jderieg Jun 2, 2023
Author

Yes, I had a typo in my group name! My bad. Thanks for the assist.

jderieg Jun 2, 2023
Author

Last issue for the day:
I can successfully create an organization, but when I create a workspace, and hit the "Create Workspace" button, it doesn't respond, and I get the following:

alfespa17 Jun 5, 2023
Maintainer

The admin group only has access to create organizations and handle team access, you will have to add specific teams to each organization and handle the permissions.

Check here:

https://docs.terrakube.org/user-guide/organizations/team-management

https://docs.terrakube.org/user-guide/workspaces/creating-workspaces

I hope this can help you.

jderieg Jun 5, 2023
Author

That worked. Thanks again. As a feature-request, it would be great if there was some pop-up text or tool-tip that gives some guidance here instead of just a dead/non-working button.
I have some additional issues now that I've at least got the app up and running. I'll start a new thread below.

alfespa17 Jun 5, 2023
Maintainer

Thank you for the feed back I have created the issue for the popup
#399

jderieg · 2023-06-05T17:21:22Z

jderieg
Jun 5, 2023
Author

I've got Terrakube up and running now with a test org and workspace, and some proper team permissions. I'm trying to migrate a state file for some terraform I currently have pointing to Terraform Cloud. Here is my cloud block:

terraform {
  cloud {
    organization = "test1"
    hostname     = "terrakube-api.<redacted>.com"
    workspaces {
      name = "test"
    }
  }
}

When I run a terraform init, I get the following error:

╷
│ Error: Host terrakube-api.<redacted>.com does not support tfe version v2
│ 
│   on workspace.tf line 25, in terraform:
│   25:     hostname     = "terrakube-api.<redacted>.com"

I also tried using a 'remote' backend instead of 'cloud' (all of our current terraform is using the 'cloud' backend), and when attempting to migrate the state, it gave me the following error:

Initializing the backend...
Migrating from Terraform Cloud to backend "remote".
Initializing modules...
╷
│ Error: Migrating state from Terraform Cloud to another backend is not yet implemented.

I'd really much rather use the 'cloud' backend anyway, but I'm not sure what that tfe error entails. I'd also really love to be able to use workspace tags instead of just the workspace name, but it doesn't seem like Terrakube has workspace tags implemented yet, correct?

7 replies

jderieg Jun 5, 2023
Author

terraform init -migrate-state                                                                                                                

Initializing the backend...
Initializing modules...
╷
│ Error: Invalid command-line option
│ 
│ The -migrate-state option is for migration between state backends only, and is not applicable when using Terraform Cloud.
│ 
│ Terraform Cloud migration has additional steps, configured by interactive prompts.

I also tried doing a 'terraform state pull ...' from the old workspace, and a 'terraform state push ...' to the new terrakube workspace. That also failed.

This is probably a deal-breaker for us, sadly. We use the cloud block with workspace tagging heavily in all of our terraform code.

jderieg Jun 5, 2023
Author

The cloud block is also the recommended method to use by Terraform:

alfespa17 Jun 5, 2023
Maintainer

We will have to check the Terraform CLI code to see how it works with the terraform "cloud backend" ( I guess it will use the terraform cloud API like the "remote backend"), maybe we just need to implement a couple of more methods to make it work.

It looks like for now if you want to migrate from terraform cloud you could only migrate if you are using the "remote backend"

Reference: #369

Regarding the question about "tags", Terrakube does not support "tags" this is something that we can add later.

Feel free to open the feature request in our main repository.

https://github.com/AzBuilder/terrakube/issues

alfespa17 Jun 5, 2023
Maintainer

By the way all help is welcome if someone would like to contribute with the project adding features or suggesting ideas to make terrakube better over time

jderieg Jun 5, 2023
Author

Thanks, I opened a new issue with the feature request: #400

jsidney · 2023-10-17T06:15:41Z

jsidney
Oct 17, 2023

Hi guys. i know this issue is a bit stale and my issue might be very different to the above.... when I built my eks cluster using eksctl (and most of the config was the default and not secure lol), I was able to launch terrakube with ease. i have tried to refine my cluster to make it a bit secure and best practice (deploying it now with terraform etc) but my pods for the executor, API are failing cause Redis isn't starting up. has anyone had this problem before?

2023-10-17 06:14:54.495  WARN 1 --- [io-8080-exec-10] o.s.b.a.redis.RedisHealthIndicator       : Redis health check failed

org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

I'm thinking it could be my security group for the nodes..... would it require a specific redis port?

0 replies

jsidney · 2023-10-17T06:38:36Z

jsidney
Oct 17, 2023

nevermind..... the redis pod is failing cause my cluster didn't install the ebs-csi add-on 😖

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Terrakube

Terrakube on EKS with nginx-ingress #397

{{title}}

Replies: 5 comments 25 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Terrakube

Terrakube on EKS with nginx-ingress #397

jderieg Jun 1, 2023

Replies: 5 comments · 25 replies

alfespa17 Jun 2, 2023 Maintainer

jderieg Jun 2, 2023 Author

jderieg Jun 2, 2023 Author

alfespa17 Jun 2, 2023 Maintainer

alfespa17 Jun 2, 2023 Maintainer

jderieg Jun 2, 2023 Author

alfespa17 Jun 2, 2023 Maintainer

jderieg Jun 2, 2023 Author

alfespa17 Jun 2, 2023 Maintainer

jderieg Jun 2, 2023 Author

jderieg Jun 2, 2023 Author

jderieg Jun 2, 2023 Author

alfespa17 Jun 5, 2023 Maintainer

jderieg Jun 5, 2023 Author

alfespa17 Jun 5, 2023 Maintainer

jderieg Jun 5, 2023 Author

jderieg Jun 5, 2023 Author

jderieg Jun 5, 2023 Author

alfespa17 Jun 5, 2023 Maintainer

alfespa17 Jun 5, 2023 Maintainer

jderieg Jun 5, 2023 Author

jsidney Oct 17, 2023

jsidney Oct 17, 2023

jderieg
Jun 1, 2023

Replies: 5 comments 25 replies

alfespa17
Jun 2, 2023
Maintainer

jderieg Jun 2, 2023
Author

jderieg Jun 2, 2023
Author

alfespa17 Jun 2, 2023
Maintainer

alfespa17 Jun 2, 2023
Maintainer

jderieg Jun 2, 2023
Author

alfespa17 Jun 2, 2023
Maintainer

jderieg Jun 2, 2023
Author

alfespa17 Jun 2, 2023
Maintainer

jderieg
Jun 2, 2023
Author

jderieg Jun 2, 2023
Author

jderieg Jun 2, 2023
Author

alfespa17 Jun 5, 2023
Maintainer

jderieg Jun 5, 2023
Author

alfespa17 Jun 5, 2023
Maintainer

jderieg
Jun 5, 2023
Author

jderieg Jun 5, 2023
Author

jderieg Jun 5, 2023
Author

alfespa17 Jun 5, 2023
Maintainer

alfespa17 Jun 5, 2023
Maintainer

jderieg Jun 5, 2023
Author

jsidney
Oct 17, 2023

jsidney
Oct 17, 2023