Controlling parallelism of pipelines

[jerriep]

We are running into an issue where we are running out of connections to our database. The specific error we are seeing is The request limit for the database is 90 and has been reached.

What happens is that we have custom shortcodes that make queries to the database. These shortcodes are included in the document content of which there are a large number of documents (200+). When the RenderRazor module runs, it seems to process these pages - and therefore the shortcodes - almost all at once and hits the database with a huge number of queries, resulting in the database running out of connections. The database in question is on Azure and the limit of 90 connections cannot be changed.

We will probably need to rethink how we do some things and try and pre-load the data required by the shortcodes so they don't do such a large number of queries, but for now, I just want to see if there are perhaps some switches we can toggle to "tone things down" a bit and get the Razor pages module to perhaps process each page in serial.

I have tried the --serial CLI parameter but that only seems to run the various pipelines themselves in serial. The razor module itself still seems to try and process all the pages in parallel, so it does not resolve the issue.

Is there anything you can advise us to do - other than rearchitecting our pipelines and loading data upfront?

[daveaglick]

I can think of a couple things you could try, but you’re correct that the optimal solution would be to pre-fetch data where you can really tailor the throttling. A lot of the modules have been designed to rip through work as fast as the internal Task thread pool will allow (independent of inter-pipeline parallelism as you noticed).

For reference, the RenderRazor module uses this ParallelSelectAsync extension which boils down to a bunch of Task.Run() wrapped in a Task.WhenAll().

So here’s some options:

• Add some throttling directly to the shortcode. The easiest way might be to use a locking primitive like a SemaphoreSlim. You should be okay on deadlock problems since Razor rendering doesn’t access or block across pages (but can’t say 100% for sure there’ll be no deadlocking - would have to try it out).
• Create a service that manages the connections and have the shortcode make calls through that. Then the service can use something like Polly which does a nice job of cleanly handling throttling scenarios.

Let me know if you need more clarification on either of these ideas, and I’ll let you know if I come up with anything else.

[jerriep]

Thank you, Dave. We'll talk these options over on our side and see how we can resolve it.

[jerriep]

@daveaglick As a follow-up, we managed to come up with a solution that seems to be working without requiring a lot of code changes. Previously, we defined the offending pipeline by inheriting from the Pipeline class and then defining the dependencies and various modules for the pipeline.

I looked at the overloads available on the Bootstrapper.AddPipeline() method and noticed that we can also define a pipeline dynamically at runtime by using one of these overloaded methods (we ended up using this one)

With that in mind, we split up the processing of the pages into batches of 50 items and added a separate pipeline for each batch. Then, we combined this with the --serial parameter for the CLI to ensure that we run the batches in serial, and that resolved the problem of overloading our database with too many queries.

[daveaglick]

Great! Glad you were able to figure out a way around it. Just let me know if you find any upstream features that might be helpful down the road - no promises, but we can at least open a proper issue to track them.