PSParallelPipeline

_{Parallel processing of pipeline input objects!}

PSParallelPipeline is a PowerShell module featuring the Invoke-Parallel cmdlet, designed to process pipeline input objects in parallel using multithreading. It mirrors the capabilities of ForEach-Object -Parallel from PowerShell 7.0+, bringing this functionality to Windows PowerShell 5.1, surpassing the constraints of Start-ThreadJob.

Why Use This Module?

Except for -AsJob, Invoke-Parallel delivers the same capabilities as ForEach-Object -Parallel and adds support for Common Parameters—a feature missing from the built-in cmdlet. For larger datasets or time-intensive tasks, Invoke-Parallel can significantly reduce execution time compared to sequential processing.

Streamlined Pipeline Processing

Invoke-Parallel can stream objects as they complete, a capability shared with ForEach-Object -Parallel. Each iteration sleeps for 1 second, but Select-Object -First 1 stops the pipeline after the first object is available, resulting in a total time of ~1 second instead of 11 seconds if all 0..10 were processed sequentially.

Measure-Command {
    $null | Invoke-Parallel { 0..10 | ForEach-Object { Start-Sleep 1; $_ } } |
        Select-Object -First 1
} | Select-Object TotalSeconds

# TotalSeconds
# ------------
#        1.06

Common Parameters Support

Unlike ForEach-Object -Parallel (up to v7.5), Invoke-Parallel supports Common Parameters, enhancing control and debugging.

# Stops on first error
0..5 | Invoke-Parallel { Write-Error $_ } -ErrorAction Stop
# Invoke-Parallel: 0

# Stops on warnings
0..5 | Invoke-Parallel { Write-Warning $_ } -WarningAction Stop
# WARNING: 1
# Invoke-Parallel: The running command stopped because the preference variable "WarningPreference" is set to Stop: 1

# Pipeline variable support
0..5 | Invoke-Parallel { $_ * 2 } -PipelineVariable pipe | ForEach-Object { "[$pipe]" }
# [6] [0] [8] [2] [4] [10]

Cleaner Timeout Handling

Get a single, friendly timeout message instead of multiple errors:

0..10 | Invoke-Parallel { $_; Start-Sleep 5 } -TimeoutSeconds 2
# 0 1 2 3 4
# Invoke-Parallel: Timeout has been reached.

$using: Scope Support

Easily pass variables into parallel scopes with the $using: modifier, just like ForEach-Object -Parallel:

$message = 'world!'
'hello ' | Invoke-Parallel { $_ + $using:message }
# hello world!

-Variables, -Functions, -ModuleNames, and -ModulePaths Parameters

• -Variables Parameter: Pass variables directly to parallel runspaces.

  'hello ' | Invoke-Parallel { $_ + $msg } -Variables @{ msg = 'world!' }
  # hello world!

• -Functions Parameter: Use local functions in parallel scopes without redefining them.

  function Get-Message {param($MyParam) $MyParam + 'world!' }
  'hello ' | Invoke-Parallel { Get-Message $_ } -Functions Get-Message
  # hello world!

• -ModuleNames Parameter: Import system-installed modules into parallel runspaces by name, using modules discoverable via $env:PSModulePath.

  Import-Csv users.csv | Invoke-Parallel { Get-ADUser $_.UserPrincipalName } -ModuleNames ActiveDirectory
  # Imports ActiveDirectory module for Get-ADUser

• -ModulePaths Parameter: Import custom modules from specified directory paths into parallel runspaces.

  $moduleDir = Join-Path $PSScriptRoot "CustomModule"
  0..10 | Invoke-Parallel { Get-CustomCmdlet } -ModulePaths $moduleDir
  # Imports custom module for Get-CustomCmdlet

These parameters are a quality-of-life enhancement, especially -Functions, which incorporates locally defined functions to the runspaces’ Initial Session State—a feature absent in ForEach-Object -Parallel and a far better option than passing function definitions into the parallel scope. The new -ModuleNames and -ModulePaths parameters simplify module integration by automatically loading system-installed and custom modules, respectively, eliminating the need for manual Import-Module calls within the script block.

Documentation

Explore detailed usage in the docs.

Installation

PowerShell Gallery

The module is available through the PowerShell Gallery:

Install-Module PSParallelPipeline -Scope CurrentUser

From Source

git clone 'https://github.com/santisq/PSParallelPipeline.git'
Set-Location ./PSParallelPipeline
./build.ps1

Requirements

• Compatible with Windows PowerShell 5.1 and PowerShell 7+
• No external dependencies

Contributing

Contributions are more than welcome! Fork the repo, make your changes, and submit a pull request. Check out the source for more details.