Dagster on AWS ECS/Fargate

[langsharpe]

I'm looking at deploying Dagster on an ECS/Fargate cluster on AWS.

I'm concerned about which Run Launcher should be used. ECSRunLauncher is under development. Would the DockerRunLauncher work? Or could I use DefaultRunLauncher which (I think) would run everything in the dagster-daemon container? That may be ok for our workload until ECSRunLauncher released.

What is your suggested configuration of Dagster on ECS/Fargate?

Thank you!

[jmsanders]

Since this question was initially posted, we've released an experimental version of the EcsRunLauncher (first available in 0.11.15) as well as a reference deployment for running Dagster on ECS.

In the absence of those two tools, you can still deploy to ECS. A popular solution is to use the DefaultRunLauncher which will launch pipelines in the daemon container like you anticipated. I'm also generally aware that some people have written their own custom EcsRunLaunchers but to date, none have been pull requested back against the project that I'm aware of.