Increase database pooling

[jvanbaarsen]

Currently we have a default of 5 for the database pooling in the database.yml that gets created by Chef-repo. This is very low, I suggest we up this to something like 50.

What do you think?

/cc @michiels / @joshuajansen

[michiels]

What kind of metric should we think about? Web requests or Sidekiq requests or something? What's the basis for the number. If you believe 50 is a good number, go for it! Curious to learn how you got to it :)

[michiels]

Btw, what pooling do you mean related to chef-repo?

[jvanbaarsen]

@michiels Its the pool size in database.yml. Its the amount connections that can happen to the database. For a project we are working on right now we need to have a few sidekiq connections (which is 20 by default), and now we need to lower this to 2 for the time being.

[michiels]

Ahh. You mean update the default database.yml in chef-repo to set it to database pooling. That wasn't originally clear from the issue description. Now I get it!

But what about people that don't use Sidekiq? Maybe offert this as a config option with a statement in the README that you should set it to a higher number when using Sidekiq?

[jvanbaarsen]

@michiels I don't think it can harm people when the number is a bit higher. I'm supported of using sane defaults, and lower the configuation hell.

[michiels]

Agreed! so 50 is a correct number? How many connections would a regular Web app with Sidekiq need? And is there a downside to make the pool large? I think Sidekiq defaults to 25 connections right?

[jvanbaarsen]

@michiels Another solution would be make it an ENV var, and set it to 50 by default like so:

production:
  pool: <%= ENV["DB_POOL_SIZE"] || 50 %>

Yes, sidekiq defaults to 25, I'm not really sure what the normal connection count would be. I think it depends on how many passenger workers you have. Since every worker needs a connection

[michiels]

@jvanbaarsen Right. That would have my preference. External apps like Intercity can then configure the pool based on what addons are enabled for an app. By default Passenger has about 1-3 workers for regular apps.