Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[helm] Cannot connect to External RDS instance #48854

Closed
joel-cant-wbt opened this issue Dec 9, 2024 · 7 comments
Closed

[helm] Cannot connect to External RDS instance #48854

joel-cant-wbt opened this issue Dec 9, 2024 · 7 comments
Labels

Comments

@joel-cant-wbt
Copy link

Helm Chart Version

1.2.0

What step the error happened?

On deploy

Relevant information

Hi.

Trying to get Airbyte to deploy on AWS EKS. I am using an external RDS instance for the airbyte config.



postgres:
  enabled: false  # Disables the internal Airbyte PostgreSQL database

serviceAccount:
  name: airbyte-service-role

# Global Environment Variables for Database Details
global:
  database:
    type: external
    secretName: "airbyte-config-secrets"
    host: "airbye.eu-west-2.rds.amazonaws.com"
    port: "5432"
    db: "airbyte_db"
    userSecretKey: "database-user"
    passwordSecretKey: "database-password"
  debug: true
  secretsManager:
    type: awsSecretManager
    secretName: "airbyte-config-secrets" # Name of your Kubernetes secret.
    awsSecretManager:
      region: eu-west-2
      authenticationType: credentials # Use "credentials" or "instanceProfile"
      tags: # Optional - Add tags to new secrets created by Airbyte
        - key: Brand
          value: M
        - key: Application
          value: Airbyte
        - key: Region
          value: 1
      kms: "" # Optional - ARN for KMS Decryption; leave blank if not used

metadata:
  name: airbyte-ingress
  namespace: airbyte
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}]'
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/subnets: subnet-0b0c186c90e677e46,subnet-0f3bc2bafce745f54


# Resource configuration for Airbyte
resources:
  limits:
    cpu: "2"            # Maximum CPU allocation
    memory: "4Gi"        # Maximum memory allocation
  requests:
    cpu: "1"            # Guaranteed minimum CPU allocation
    memory: "2Gi"       # Guaranteed minimum memory allocation

nodeSelector:
  eks.amazonaws.com/nodegroup: airbyte-ng

# Ingress configuration for custom domain
ingress:
  enabled: true
  hosts:
    - host: airbyte.aws.com
      paths:
        - path: /
          pathType: Prefix

but it fails to connect to the external DB. I have verified that EKS nodes can connect to the external DB with the username and password no problem.

Looks like the YAML variables are not being correctly picked up:

[cloudshell-user@ip-10-132-24-94 ~]$ kubectl exec -n airbyte -it airbyte-airbyte-bootloader -- env | grep AIRBYTE
AIRBYTE_VERSION=1.2.0
AIRBYTE_MINIO_SVC_PORT=tcp://172.20.85.226:9000
AIRBYTE_DB_SVC_PORT_5432_TCP=tcp://172.20.75.103:5432
AIRBYTE_MINIO_SVC_PORT_9000_TCP_PROTO=tcp
AIRBYTE_MINIO_SVC_PORT_9000_TCP_PORT=9000
AIRBYTE_DB_SVC_PORT_5432_TCP_PORT=5432
AIRBYTE_MINIO_SVC_SERVICE_HOST=172.20.85.226
AIRBYTE_MINIO_SVC_SERVICE_PORT=9000
AIRBYTE_DB_SVC_SERVICE_PORT=5432
AIRBYTE_DB_SVC_PORT_5432_TCP_PROTO=tcp
AIRBYTE_MINIO_SVC_PORT_9000_TCP=tcp://172.20.85.226:9000
AIRBYTE_MINIO_SVC_PORT_9000_TCP_ADDR=172.20.85.226
AIRBYTE_DB_SVC_SERVICE_HOST=172.20.75.103
AIRBYTE_DB_SVC_PORT=tcp://172.20.75.103:5432
AIRBYTE_DB_SVC_PORT_5432_TCP_ADDR=172.20.75.103

logs show

2024-12-09 15:25:32,224 [main]  ERROR   i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):78 - Failed to verify database connection.
org.jooq.exception.DataAccessException: Error getting connection from data source HikariDataSource (HikariPool-2)
        at org.jooq_3.19.7.POSTGRES.debug(Unknown Source)
        at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:90)
        at org.jooq.impl.DefaultExecuteContext.connection(DefaultExecuteContext.java:651)
        at org.jooq.impl.AbstractQuery.connection(AbstractQuery.java:388)
        at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:308)
        at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:301)
        at org.jooq.impl.AbstractResultQuery.fetchLazyNonAutoClosing(AbstractResultQuery.java:322)
        at org.jooq.impl.SelectImpl.fetchLazyNonAutoClosing(SelectImpl.java:3256)
        at org.jooq.impl.ResultQueryTrait.fetchOne(ResultQueryTrait.java:509)
        at org.jooq.impl.Tools.attach(Tools.java:1652)
        at org.jooq.impl.DefaultDSLContext.fetchOne(DefaultDSLContext.java:5019)
        at org.jooq.impl.DefaultDSLContext.lambda$fetchValue$55(DefaultDSLContext.java:5039)
        at org.jooq.impl.Tools.attach(Tools.java:1652)
        at org.jooq.impl.DefaultDSLContext.fetchValue(DefaultDSLContext.java:5039)
        at org.jooq.impl.DefaultDSLContext.fetchValue(DefaultDSLContext.java:5058)
        at org.jooq.impl.DefaultDSLContext.fetchExists(DefaultDSLContext.java:5150)
        at io.airbyte.db.check.DatabaseAvailabilityCheck.lambda$isDatabaseConnected$0(DatabaseAvailabilityCheck.java:76)
        at io.airbyte.db.Database.query(Database.java:23)
        at io.airbyte.db.check.DatabaseAvailabilityCheck.lambda$isDatabaseConnected$1(DatabaseAvailabilityCheck.java:76)
        at io.airbyte.db.check.DatabaseAvailabilityCheck.check(DatabaseAvailabilityCheck.java:47)
        at io.airbyte.db.init.DatabaseInitializer.initialize(DatabaseInitializer.java:45)
        at io.airbyte.bootloader.Bootloader.initializeDatabases(Bootloader.java:229)
        at io.airbyte.bootloader.Bootloader.load(Bootloader.java:104)
        at io.airbyte.bootloader.Application.main(Application.java:22)
Caused by: java.sql.SQLTransientConnectionException: HikariPool-2 - Connection is not available, request timed out after 30000ms (total=0, active=0, idle=0, waiting=0)
        at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:686)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:179)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:144)
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:99)
        at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:87)
        ... 22 common frames omitted
Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "airbyte_user"
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:711)
        at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:213)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:268)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:273)
        at org.postgresql.Driver.makeConnection(Driver.java:446)
        at org.postgresql.Driver.connect(Driver.java:298)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:137)
        at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:360)
        at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:202)
        at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:461)
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:724)
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:703)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

I see the failed password attempt, but it seems to be using the wrong DB.

Relevant log output

@cmanzur
Copy link

cmanzur commented Dec 9, 2024

Hello, you're missing the following secret, it needs to be created BEFORE the helm install:

apiVersion: v1
kind: Secret
metadata:
  name: airbyte-config-secrets
  namespace: airbyte  <== change this namespace
type: Opaque
stringData:
  # Database Secrets
  database-user: "database-user"
  database-password: "database-password"

Docs: https://docs.airbyte.com/deploying-airbyte/integrations/database

@joel-cant-wbt
Copy link
Author

Hello, you're missing the following secret, it needs to be created BEFORE the helm install:

apiVersion: v1
kind: Secret
metadata:
  name: airbyte-config-secrets
  namespace: airbyte  <== change this namespace
type: Opaque
stringData:
  # Database Secrets
  database-user: "database-user"
  database-password: "database-password"

Docs: https://docs.airbyte.com/deploying-airbyte/integrations/database

I have that already, secret is setup. Seems to me its still trying to connect to the local database instance.

@PrestonBlackburn
Copy link

Not sure if this helps, but I'm also having some Postgres RDS issues (outlined in this issue: #49304)

When I was having connection issues (now different issues), it was because of my security group settings for RDS were not set up correctly.

If I run kubectl describe pod on the bootloader pod I can see my env variables are comming from the airbyte-config-secrets - it was just the security group settings blocking communication.

    Environment:
      AIRBYTE_VERSION:                    <set to the key 'AIRBYTE_VERSION' of config map 'release-1-airbyte-env'>                    Optional: false
      RUN_DATABASE_MIGRATION_ON_STARTUP:  <set to the key 'RUN_DATABASE_MIGRATION_ON_STARTUP' of config map 'release-1-airbyte-env'>  Optional: false
      DATABASE_HOST:                      <set to the key 'DATABASE_HOST' of config map 'release-1-airbyte-env'>                      Optional: false
      DATABASE_PORT:                      <set to the key 'DATABASE_PORT' of config map 'release-1-airbyte-env'>                      Optional: false
      DATABASE_DB:                        <set to the key 'DATABASE_DB' of config map 'release-1-airbyte-env'>                        Optional: false
      DATABASE_USER:                      <set to the key 'database-user' in secret 'airbyte-config-secrets'>                         Optional: false
      DATABASE_PASSWORD:                  <set to the key 'database-password' in secret 'airbyte-config-secrets'>                     Optional: false
      DATABASE_URL:                       <set to the key 'DATABASE_URL' of config map 'release-1-airbyte-env'>                       Optional: false

@joel-cant-wbt
Copy link
Author

Not sure if this helps, but I'm also having some Postgres RDS issues (outlined in this issue: #49304)

When I was having connection issues (now different issues), it was because of my security group settings for RDS were not set up correctly.

If I run kubectl describe pod on the bootloader pod I can see my env variables are comming from the airbyte-config-secrets - it was just the security group settings blocking communication.

Environment:
  AIRBYTE_VERSION:                    <set to the key 'AIRBYTE_VERSION' of config map 'release-1-airbyte-env'>                    Optional: false
  RUN_DATABASE_MIGRATION_ON_STARTUP:  <set to the key 'RUN_DATABASE_MIGRATION_ON_STARTUP' of config map 'release-1-airbyte-env'>  Optional: false
  DATABASE_HOST:                      <set to the key 'DATABASE_HOST' of config map 'release-1-airbyte-env'>                      Optional: false
  DATABASE_PORT:                      <set to the key 'DATABASE_PORT' of config map 'release-1-airbyte-env'>                      Optional: false
  DATABASE_DB:                        <set to the key 'DATABASE_DB' of config map 'release-1-airbyte-env'>                        Optional: false
  DATABASE_USER:                      <set to the key 'database-user' in secret 'airbyte-config-secrets'>                         Optional: false
  DATABASE_PASSWORD:                  <set to the key 'database-password' in secret 'airbyte-config-secrets'>                     Optional: false
  DATABASE_URL:                       <set to the key 'DATABASE_URL' of config map 'release-1-airbyte-env'>                       Optional: false

Thanks - the only thing i can see is possible somehow the password is getting messed up - i can connect from inside the EKS cluster absoloutely fine

@marcosmarxm
Copy link
Member

@joel-cant-wbt did you find a solution or still having issues?

@marcosmarxm marcosmarxm changed the title Cannot connect to External RDS instance [helm] Cannot connect to External RDS instance Dec 16, 2024
@joel-cant-wbt
Copy link
Author

@joel-cant-wbt did you find a solution or still having issues?

Still having the same issue

From the logs:

Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "airbyte_user"

Password tests fine, but i also noted this error:

Caused by: org.postgresql.util.PSQLException: Connection to airbyte-db-svc:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

Which makes it look like its trying the wrong data source.

@joel-cant-wbt
Copy link
Author

i figured out what was missing.

postgres: should have been postgresql: in the yaml file

Always something simple

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants