Wanting scaling documentation

[japhb]

(The following is a documentation request, and most of the following is an extended explanation of what kind of doc I'm looking for. To be clear, I am not trying to build the thing I describe below. It's just a good model for the actual problems I want to solve because I think it would deal with similar concurrency issues, there's a whole lot of interesting knowledge that can be gained from studying it, and I believe most people would be familiar with the problem space below, thus avoiding the need for a lot of side explanations or "Why are you doing that?" derailing. I hope.)

It's difficult to tell what the true concurrency capabilities of Cro are. There is a short note at https://cro.services/docs/structuring-services#A_note_on_concurrency that explains that multiple threads can be in the routes at the same time, but there's no good overview of how one should structure a Cro app for heavy scaling -- and how to load-test it reasonably using native Raku code (as opposed to e.g. ApacheBench, for HTTP services). In addition, Cro's source and Rakudo's concurrency code both have many levels of abstraction, and it's difficult to understand how introduction of concurrency (e.g. task creation) interacts with concurrency limits (locking, blocking, and other forms of mutex) and the oodles of plumbing between to determine the actual concurrency envelope of which Cro is capable.

To give a little concreteness to this issue, imagine trying to build a server for a high-volume chat service. For scale let's think of an IRC server on a major network; at the moment my freenode IRC server is claiming ~7500 local connections, ~90K global users, ~50K channels. If I make the simplifying assumption of power-law distributions for users-per-channel, channels-per-user/bot, message-rate-per-channel, etc., pretty soon we're talking some serious concurrency. Could a Cro server scale to anywhere near that level? How could it be structured to do so? And how much compute hardware could you throw at it before Cro/Rakudo are no longer able to gain performance from more cores?

Let's add a couple more twists. First, what if the server had to do heavier work (both more compute, and more in-system latency) for each message, rather than just fanning out the incoming messages to each receiver? For instance, what if it translated every message into the preferred language of each client that received it (either using an internal library or some external service), and then logged all messages to a database in both original and translated forms? Can this extra work be added without severely limiting the server's scalability?

Second, imagine you've done everything you can think of to squeeze out the maximum scalability you can out of your server code. Now you want to test whether all that design and coding actually worked and the server will scale as advertised, before you add it into your hypothetical chat network and risk it quickly falling over under the load. How could you design test clients that could push the server to its limits, and be sure any scalability problems they expose are actually in the server and not in the clients?