Powerful Pipes - The power of UNIX Pipes

In a nutshell Powerful Pipes is a library for working with UNIX Pipes.

Install

> pip install powerful-pipes

Quick Start

Create a CLI tool that reads from the JSON from stdin and dumps to the stdout, after processing input data:

# File: pipe-example.py
from powerful_pipes import read_json_from_stdin, eprint, write_json_to_stdout

for error, input_json in read_json_from_stdin():

    if error:
        # Here you can manager the error. Most common error is that the
        # input is not a valid json.
        # eprint(...) function dumps the error message to the stderr
        eprint(message="Error processing input from stdin")
        continue

    try:
        input_json["myData"] = "data 1"

    except Exception as e:
        report_exception(input_json, e)

    finally:
        # THIS STEP IS CRITICAL. If you don't put again in the stdout the
        # input data, following tools in the pipe chain wouldn't receive
        # the data
        write_json_to_stdout(input_json)

Using in CLI:

> echo '{}' | python pipe-example.py | jq
{
    "myData": "data 1"
}

Advanced Usage

If you need to pass heavy data between tools in the pipe chain, you should use the write_to_stdout_by_file_ref and read_from_stdin_by_file_ref.

It stores the data in a temporary file and passes the file reference to the next tool in the pipe chain.

# File: pass_by_reference.py
from powerful_pipes import read_from_stdin_by_file_ref, eprint, write_to_stdout_by_file_ref, report_exception

for error, input_json in read_from_stdin_by_file_ref():

    if error:
        # Here you can manager the error. Most common error is that the
        # input is not a valid json.
        # eprint(...) function dumps the error message to the stderr
        eprint(message="Error processing input from stdin")
        continue

    try:
        input_json["myData"] = "data 1"

    except Exception as e:
        report_exception(input_json, e)

    finally:
        # THIS STEP IS CRITICAL. If you don't put again in the stdout the
        # input data, following tools in the pipe chain wouldn't receive
        # the data
        write_to_stdout_by_file_ref(input_json)

Changelog

You can check Changelog.

Documentation

You can find the complete documentation at: Documentation

Authors

Powerful Pipes was made by 42Crunch Research Team:

• jc42
• cr0hn

License

Powerful Pipes is Open Source and available under the Apache 2.

Contributions

Contributions are very welcome. See CONTRIBUTING.md or skim existing tickets to see where you could help out.

Acknowledgements

Special thanks to Cesar Gallego for his mentoring in data processing, that inspired this project.

Project logo thanks to Pipe icons created by srip - Flaticon