allow parallel dependency processing.

[pseudomorph]

Description

Fixes #232

Allows optional parallel processing of dependencies.

TODOs

Read the Gruntwork contribution guidelines.

• Update the docs.
• Run the relevant tests successfully, including pre-commit checks.
• Ensure any 3rd party code adheres with our license policy or delete this line if its not applicable.
• Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes (draft)

Added optional parallel processing of dependencies via --parallel-for-each flag to improve performance when processing multiple dependency iterations.

Migration Guide

[pseudomorph]

Not sure what makes sense for flag name. Please let me know if this should be changed.

[yhakbar]

This should be inverted. To be consistent with how Terragrunt works, there should be a --parallelism flag that allows users to opt out of parallel processing by setting the size of the semaphore to 1.

[pseudomorph]

@yhakbar - Thanks. My concern with introducing it as opt-out would be the implications for unmasking contention for boilerplate templates which might currently be operating within the same dir (shell/hooks). But I suppose this would be less problematic if released as a breaking change with sufficient documentation.

I can get an inversion pushed out sometime this week.

[pseudomorph]

I've focused a bit on implementing caching of redundant operations. Initial testing has shown huge efficiency gains (rendering 1000s of iterative dependencies in a matter of seconds). Parallelism becomes less important for pure templating, but still shows use when templates shell out.

I'll likely combine the caching effort here.