[config-resource] Configuring Connector-Specific not working

[shubhamqua]

Helm Chart Version

0.49.4

What step the error happened?

Other

Revelant information

I have updated resource_requirements in the actor_definition table with cpu and memory limits and requests, However, i am still seeing discrepancies in the logs regarding the cpu and memory requests and limits. I updated the actor_definiton with following values:
"cpu_limit": "0.5", "cpu_request": “0.1”, "memory_limit": "500Mi", "memory_request": "500Mi"

But in the logs i see the following CPU and memory limits for the same connector:
cpuRequest=0.1,cpuLimit=1,memoryRequest=25Mi,memoryLimit=50Mi

I also confirmed resource_requirements in the connection table is null hence that could not be interfering with this config.

I have airbyte deployed on EKS and version of airbyte being used is: 0.50.31.

Relevant log output

2024-01-02 12:21:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-01-02 12:21:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-01-02 12:21:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-01-02 12:21:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-01-02 12:21:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-01-02 12:21:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-01-02 12:21:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-01-02 12:21:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-01-02 12:21:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2024-01-02 12:21:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-01-02 12:21:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2024-01-02 12:21:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-01-02 12:21:39 INFO i.a.w.p.KubeProcessFactory(create):128 - Attempting to start pod = destination-snowflake-write-2580-0-mplhj for airbyte/destination-snowflake:3.4.2 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@45374e20[cpuRequest=0.2,cpuLimit=1,memoryRequest=1Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@3d02114c[cpuRequest=0.05,cpuLimit=0.2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@59a1716[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@1cb5507a[cpuRequest=0.1,cpuLimit=1,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@630c5b82[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts null
2024-01-02 12:21:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2024-01-02 12:21:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2024-01-02 12:21:39 INFO i.a.w.p.KubeProcessFactory(create):132 - destination-snowflake-write-2580-0-mplhj stdoutLocalPort = 9016
2024-01-02 12:21:39 INFO i.a.w.p.KubeProcessFactory(create):128 - Attempting to start pod = source-google-analytics-data-api-read-2580-0-wwnmv for airbyte/source-google-analytics-data-api:2.0.1 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@57c7ef2c[cpuRequest=0.2,cpuLimit=1,memoryRequest=1Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@3d02114c[cpuRequest=0.05,cpuLimit=0.2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@77d388dc[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=null, stdOut=io.airbyte.config.ResourceRequirements@4893946b[cpuRequest=0.2,cpuLimit=1,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts io.airbyte.config.AllowedHosts@53b000c[hosts=[oauth2.googleapis.com, www.googleapis.com, analyticsdata.googleapis.com, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2024-01-02 12:21:39 INFO i.a.w.p.KubeProcessFactory(create):135 - destination-snowflake-write-2580-0-mplhj stderrLocalPort = 9017
2024-01-02 12:21:39 INFO i.a.w.p.KubeProcessFactory(create):132 - source-google-analytics-data-api-read-2580-0-wwnmv stdoutLocalPort = 9018
2024-01-02 12:21:39 INFO i.a.w.p.KubeProcessFactory(create):135 - source-google-analytics-data-api-read-2580-0-wwnmv stderrLocalPort = 9019
2024-01-02 12:21:39 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):668 - Creating stdout socket server...
2024-01-02 12:21:39 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):668 - Creating stdout socket server...
2024-01-02 12:21:39 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$11):686 - Creating stderr socket server...
2024-01-02 12:21:39 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$11):686 - Creating stderr socket server...
2024-01-02 12:21:39 INFO i.a.w.p.KubePodProcess(<init>):593 - Creating pod source-google-analytics-data-api-read-2580-0-wwnmv...</init>
2024-01-02 12:21:39 INFO i.a.w.p.KubePodProcess(<init>):593 - Creating pod destination-snowflake-write-2580-0-mplhj...</init>
2024-01-02 12:21:39 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):350 - Waiting for init container to be ready before copying files...
2024-01-02 12:21:39 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):350 - Waiting for init container to be ready before copying files...
2024-01-02 12:21:39 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):354 - Init container present..
2024-01-02 12:21:39 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):354 - Init container present..
2024-01-02 12:21:41 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):357 - Init container ready..
2024-01-02 12:21:41 INFO i.a.w.p.KubePodProcess(<init>):624 - Copying files...</init>
2024-01-02 12:21:41 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):299 - Uploading file: destination_config.json
2024-01-02 12:21:41 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):307 - kubectl cp /tmp/6ccdc861-9077-4c89-9093-1a5c39e1808b/destination_config.json airbyte-dev/destination-snowflake-write-2580-0-mplhj:/config/destination_config.json -c init
2024-01-02 12:21:41 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):357 - Init container ready..
2024-01-02 12:21:41 INFO i.a.w.p.KubePodProcess(<init>):624 - Copying files...</init>

[BravoDeltaBD]

I am encountering the exact same issue on our AKS clusters with helm chart 49.1 and 53.120. Is there any timeline on when this issue can be looked at?

[learningnumberone]

I also encountered this issue. Does anybody know a fix or workaround? We are using Airbyte version 50.56
We set resource_requirements on connections as workaround, but is far from ideal for our use case.

[intelc]

We have the same probelm in Helm 0.64.205 and version 0.57.3. The connector level setting doesn't work. However, connection level works