IAM user credentials for on-prem k8s cluster with S3 storage & RDS database

[djbutler]

I'm trying to deploy Flyte to an on-prem k8s cluster (microk8s), but with storage and databases in the cloud (AWS RDS and S3). I created an IAM user and gave the credentials to Flyte via helm values.yml:

configuration:
  database:
    username: flyte
    password: <hidden>
    host: <my rds host>
    port: 5432
    dbname: app 
    options: "" # this is needed in order to use SSL
  storage:
    metadataContainer: <my-bucket>
    userDataContainer: <my-bucket>
    provider: s3
    providerConfig:
      s3:
        region: "us-west-2"
        authType: "accesskey"
        accessKey: "<my access key>"
        secretKey: "<my secret key>"
  inline:
    plugins:
      k8s:
        inject-finalizer: true
        default-env-vars:
          - AWS_METADATA_SERVICE_TIMEOUT: 5
          - AWS_METADATA_SERVICE_NUM_ATTEMPTS: 20
    storage:
      cache:
        max_size_mbs: 100
        target_gc_percent: 100

microk8s helm install flyte-backend flyteorg/flyte-binary --namespace flyte --values values.yml

But now I have a strange situation - it seems that only some of the Flyte processes or containers can access the AWS resources. The workflows are created, and executions are initiated. Things show up in S3 and RDS. Also, tasks defined using ContainerTask seem to succeed (https://docs.flyte.org/projects/cookbook/en/v0.3.66/auto/core/containerization/raw_container.html). But raw python tasks (created with the @task decorator) fail with this error:

Unable to locate credentials

So it seems that perhaps the AWS credentials are not getting passed through to all containers - in particular, to the default container. What can I do?

[kumare3]

Raw python tasks should use a the same config

[djbutler]

Thanks - I'm not sure what the problem is then. I solved it by adding AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to inline.plugins.k8s.default-env-vars