Resource connection pooling

[derekseymour]

Hi. Is there a way to pool connections to resources (MySQL, Postgres, etc) on scripts and flows? Architecturally, does Windmill handle anything along these lines in terms of Resources? If not, would it be possible, for instance, for scripts to have warmups/cooldowns where execution on new workers could pre-run and establish connections? Or, have some sort of long running script that could manage the connection pool on each worker? (just thinking out loud :)

Thanks for the amazing work you continue to put into this project!

[rubenfiszel]

Thank you for the suggestions and thoughts.

For doing anything related to connection pool management, we would have to handle them as "native jobs" like postgresql and soon mysql is (the connection and query execution is handled by the engine backend).

Currently, any SQL requests done as a native job is done with a new connection that is aborted at the end of the query. It's costly but it's better than maintaining a connection/pool since most queries are really low throughput.

However, in most cases, one workspace will have a few high throughput databases for which maintaining a connection pool would make sense.

I think the best way to do this is to have an array of instance wide connection pool set at the worker level configuration that when matching a resource is used instead of establishing a connection. This would work well with our dedicated enterprise plans and self-hosted setups.