Skip to content

Latest commit

 

History

History
34 lines (23 loc) · 1.24 KB

README.md

File metadata and controls

34 lines (23 loc) · 1.24 KB
Raw

Blockpype

Breaks apart input stream into blocks and pipes each block into newly spawned processes

Installing

pip install git+https://github.com/kfzteile24/blockpype.git

Testing

Clone the repository and run visual tests that display each started PID and their output:

  • test/by-byte.sh - should break apart in the middle of the 2-byte character
  • test/by-char.sh - should break apart in the middle of some numbers
  • test/by-line.sh - should break apart nicely line by line

Usage

cat content.txt | blockpype -l 10 other-program --arg-for-program --another-arg "argument value"

It will cut content.txt into blocks of 10 lines, and for each block, a process will be started using the command:

other-program --arg-for-program --another-arg "argument value"

and the contents of the block will be served to stdin

What's it for?

Whenever you want to break apart a large stream when the other-program consumer stores it into memory until EOF, and you don't have enough memory for it.

Similar to split --filter "other-program --arg-for-program --another-arg \"argument value\"" except less clunky, other-program gets launched from current pwd with the current user, and with the current environment variables.