ref(tokio): Limit blocking tokio pool to 150 threads (#3840)

We only have a single usage of `spawn_blocking` left in Relay, it's the
fetching of project configs from Redis, these are limited by the amount
of connections in the Redis pool anyways (much lower than 150).

We've also seen the pool explode at runtime up to the maximum and never
shrink again which causes a huge amount of wasted RAM (and CPU?).

This PR limits the pool to 150 threads (which should be still wayyyy to
much) and at the same time lowers the keep alive to encourge the runtime
to give up threads more easily again.

relay-server/src/service.rs

@@ -1,6 +1,7 @@
 use std::convert::Infallible;
 use std::fmt;
 use std::sync::Arc;
+use std::time::Duration;
 
 use crate::metrics::{MetricOutcomes, MetricStats};
 use crate::services::stats::RelayStats;
@@ -77,6 +78,20 @@ pub fn create_runtime(name: &str, threads: usize) -> Runtime {
     tokio::runtime::Builder::new_multi_thread()
         .thread_name(name)
         .worker_threads(threads)
+        // Relay uses `spawn_blocking` only for Redis connections within the project
+        // cache, those should never exceed 100 concurrent connections
+        // (limited by connection pool).
+        //
+        // Relay also does not use other blocking opertions from Tokio which require
+        // this pool, no usage of `tokio::fs` and `tokio::io::{Stdin, Stdout, Stderr}`.
+        //
+        // We limit the maximum amount of threads here, we've seen that Tokio
+        // expands this pool very very aggressively and basically never shrinks it
+        // which leads to a massive resource waste.
+        .max_blocking_threads(150)
+        // As with the maximum amount of threads used by the runtime, we want
+        // to encourage the runtime to terminate blocking threads again.
+        .thread_keep_alive(Duration::from_secs(1))
         .enable_all()
         .build()
         .unwrap()