Syncs don't respect the resource_requirements set in the actor_definition table in the database in latest Airbyte

[ro8inmorgan]

Helm Chart Version

0.67.17

What step the error happened?

During the Sync

Relevant information

I've set the resource_requirements for my connectors in the postgress database in table actor_definition. For example

{"jobSpecific": [{"jobType": "sync", "resourceRequirements": {"cpu_limit": "1", "cpu_request": "0.5", "memory_limit": "1Gi", "memory_request": "1Gi"}}]}

But all syncs revert to the default settings set in the airbyte-env configmap

No matter what I do the actor_defintion resource_requirements is never used. I would like to set a per connector resource base as these widely differ per connector.

For example logs below come from a connector set with the resource_requirements above as you can see it doesn't follow the resource_requirements but uses the settings from airbyte-env configmap which are:

JOB_MAIN_CONTAINER_CPU_LIMIT: "2"
JOB_MAIN_CONTAINER_CPU_REQUEST: "1"
JOB_MAIN_CONTAINER_MEMORY_LIMIT: 2Gi
JOB_MAIN_CONTAINER_MEMORY_REQUEST: 2Gi

Relevant log output

2024-05-14 07:09:42 replication-orchestrator > Attempting to start pod = source-mssql-read-102432-1-sywjb for airbyte/source-mssql:2.0.1 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@6e986dbf[cpuRequest=1,cpuLimit=2,memoryRequest=2Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@440bf54d[cpuRequest=0.05,cpuLimit=0.2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@5c7a9a3e[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=null, stdOut=io.airbyte.config.ResourceRequirements@3dc75835[cpuRequest=0.5,cpuLimit=1,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts io.airbyte.config.AllowedHosts@752ffce3[hosts=[192.168.240.5, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2024-05-14 07:09:42 replication-orchestrator > Attempting to start pod = destination-bigquery-write-102432-1-busrh for airbyte/destination-bigquery:2.4.19 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@7761bb99[cpuRequest=1,cpuLimit=2,memoryRequest=2Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@440bf54d[cpuRequest=0.05,cpuLimit=0.2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@4cb85b24[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@389b79d5[cpuRequest=0.5,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@42dcb02c[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts nul

[intelc]

We have the same probelm in Helm 0.64.205 and version 0.57.3. The connector level setting doesn't work. However, connection level works

[NAjustin]

We're still seeing the same thing when trying to override on the connector level (Helm chart version 0.293.4/app version 0.63.8).

[descampsk]

Unable to override at the actor definition level in helm 0.453.2 and airbyte 0.63.19.

Here is my SQL request to update resourceRequirements:

UPDATE
  "public"."actor_definition"
SET
  resource_requirements = '{"jobSpecific": [{"jobType": "sync", "resourceRequirements": {"cpu_limit": "4", "cpu_request": "0.1", "memory_limit": "2Gi", "memory_request": "1Gi"}}]}'
WHERE
  id IN (
  SELECT
    actor_definition_id
  FROM
    "public"."actor_definition_version"
  WHERE
    docker_repository LIKE 'airbyte/destination-bigquery' );

I need to do it on actor definition level because bigquery destination does not need much cpu but orchestrator and read pods do need much more.

[XIAZY]

Can confirm the issue for version 0.58.0. We are trying to set different resource requirements for sync and check/discover pods since they require different amount of resources, but unsuccessful

[kev-datams]

hello, is this issue still observed ?

[descampsk]

In my opinion, yes.

I could run additionnel test to be sure but I don't see any changes in the requirements, so it should still be there.

[kev-datams]

Ok thanks 🙏