Dagster k8s-celery executor

[hajiuduman-licent]

Hi,

We are developing a data quality software, where we have multiple data processing stages. We are planning to execute each stage as separate celery-k8s executor. We are using helm chart with below values.

dagster-user-deployments:
  deployments:
    - name: "k8s-example-user-code-1"
           image:             
              repository: "cusom/repo.image"
runLauncher:
  type: 
     celeryK8sRunLauncher

Repositories are loaded from our custom user code deployment image.

Executor configuration in UI

execution:
    celery-k8s:
      config:
        job_image: 'my_repo.com/stage1:latest'

But while running the pipeline we are getting below error message from k8s (my_repo.com/stage1 pod) logs.

Error 1: Post processing at path root:instance_config_map of original value {'env': 'DAGSTER_K8S_INSTANCE_CONFIG_MAP'} failed:
dagster.config.errors.PostProcessingError: You have attempted to fetch the environment variable "DAGSTER_K8S_INSTANCE_CONFIG_MAP" which is not set. In order for this execution to succeed it must be set in this environment.

Stack Trace:
  File "/usr/local/lib/python3.9/site-packages/dagster/config/post_process.py", line 77, in _post_process
    new_value = context.config_type.post_process(config_value)
  File "/usr/local/lib/python3.9/site-packages/dagster/config/source.py", line 42, in post_process
    return str(_ensure_env_variable(cfg))
  File "/usr/local/lib/python3.9/site-packages/dagster/config/source.py", line 16, in _ensure_env_variable
    raise PostProcessingError(

Error 2: Post processing at path root:postgres_password_secret of original value {'env': 'DAGSTER_K8S_PG_PASSWORD_SECRET'} failed:
dagster.config.errors.PostProcessingError: You have attempted to fetch the environment variable "DAGSTER_K8S_PG_PASSWORD_SECRET" which is not set. In order for this execution to succeed it must be set in this environment.

Stack Trace:
  File "/usr/local/lib/python3.9/site-packages/dagster/config/post_process.py", line 77, in _post_process
    new_value = context.config_type.post_process(config_value)
  File "/usr/local/lib/python3.9/site-packages/dagster/config/source.py", line 42, in post_process
    return str(_ensure_env_variable(cfg))
  File "/usr/local/lib/python3.9/site-packages/dagster/config/source.py", line 16, in _ensure_env_variable

Can you please help us in solving this issue?

[alangenfeld]

dupe of #4282