PaSh: Light-touch Data-Parallel Shell Processing

A system for parallelizing POSIX shell scripts. Hosted by the Linux Foundation.

Service	Master	Develop
Tests
Build
Pages

Quick Jump: Running PaSh | Installation | Testing | Repo Structure | Community & More | Citing

Running PaSh

To parallelize, say, ./evaluation/intro/hello-world.sh with parallelization degree of 2× run:

./pa.sh ./evaluation/intro/hello-world.sh

Run ./pa.sh --help to get more information about the available commands. Jump to docs/tutorial for a longer tutorial.

Installation

On Ubuntu, Fedora, Debian, or Arch, run curl up.pash.ndr.md | sh to set up PaSh.

For more details, manual installation, or other platforms see installation instructions.

Testing

To execute the current tests before committing and pushing code, simply run:

./scripts/run_tests.sh

Repo Structure

This repo hosts the core pash development. The structure is as follows:

• annotations: DSL characterizing commands, parallelizability study, and associated annotations.
• compiler: Shell-dataflow translations and associated parallelization transformations.
• docs: Design documents, tutorials, installation instructions, etc.
• evaluation: Shell pipelines and example scripts used for the evaluation.
• runtime: Runtime component — e.g., eager, split, and assocaited combiners.
• scripts: Scripts related to continuous integration, deployment, and testing.

Community & More

Chat:

• Discord Server (Invite)

Mailing Lists:

• pash-dev: Join this mailing list for discussing all things pash
• pash-commits: Join this mailing list for commit notifications

Development/contributions:

• Contribution guide: docs/contributing
• Continuous Integration Server: ci.binpa.sh

Citing

If you used PaSh, consider citing the following paper:

@inproceedings{pash2021eurosys,
author = {Vasilakis, Nikos and Kallas, Konstantinos and Mamouras, Konstantinos and Benetopoulos, Achilles and Cvetkovi\'{c}, Lazar},
title = {PaSh: Light-Touch Data-Parallel Shell Processing},
year = {2021},
isbn = {9781450383349},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3447786.3456228},
doi = {10.1145/3447786.3456228},
pages = {49–66},
numpages = {18},
keywords = {POSIX, Unix, pipelines, automatic parallelization, source-to-source compiler, shell},
location = {Online Event, United Kingdom},
series = {EuroSys '21}
}