Automatically remove Containers with DockerRunLauncher

[PascalStehling]

Hi,

I have the following problem. I have a Dagster Setup running with the DockerRunLauncher. Everything is working fine. The Problem is, that for each Job, a container is created, which is not deleted afterwards. As each container is multiple Gigabytes big (because of some Data Wrangling) the Hard-Drive is filled very fast, which leads to crashing the Database because there is not enough Hard-Drive space left.

The Question: Is there a way to setup the DockerRunLauncher in a way, that after the container ran and did its job, its automatically removed?

Thanks for your help in advance.

[johannkm]

Yes:

# dagster.yaml
run_launcher:
  class: DockerRunLauncher
  module: dagster_docker
  config:
    container_kwargs:
      auto_remove: true

This will get passed to the the auto_remove param of the Docker api run method, documented here: https://docker-py.readthedocs.io/en/stable/containers.html#docker.models.containers.ContainerCollection.run

[johannkm]

The only downside is that containers get removed immediately, so you'll likely turn it off if you need to do some debugging. I wish there were a TTL option.

[geekodour]

This was working till last week with the dagster docker agent, suddenly stopped working.

File "/dagster/dagster/_core/instance/config.py", line 137, in dagster_instance_config                                                                                                                                                                                              │
│       raise DagsterInvalidConfigError(                                                                                                                                                                                                                                                  │
│   dagster._core.errors.DagsterInvalidConfigError: Errors whilst loading dagster instance config at dagster.yaml.                                                                                                                                                                        │
│       Error 1: Received unexpected config entry "container_kwargs" at path root:user_code_launcher. Expected: "{ class: String config?: { } module: String }".

Any change that affected this? @johannkm

[geekodour]

I am using the docker.io/dagster/dagster-cloud-agent:latest image