Ability to pipe to cmd2 commands

[tleonhardt]

One feature of a "normal" shell that cmd2 applications are currently missing is the ability for a cmd2 command to pipe output to other cmd2 commands - it is limited to piping output to shell commands only. Occasionally this limitation feels significant. I know it can be worked around by creating a custom Python script and using the run_pyscript command to run that, but it would be much more natural if we could just pipe output to other cmd2 commands.

I have no idea how easy or difficult implementing this feature might be. But we should at least explore the possibility.

This would be a great "big new feature" to include in a 2.0 release.

[kmvanbrunt]

I think we'll have to use something like a double pipe || to distinguish between piping output to a built-in command and an OS shell command.

Lots of thoughts come to mind for a change like this.

1. Command line parsing will have to split built-in commands using ||. It will also have to resolve aliases for each command.

2. What will a Statement contain? Just one of the commands or all of them? If only one of the commands, that may mess up history items.

3. Tab completion will need to be aware of || to know when a new command starts. It already handles |, >, and >>.

4. What does it mean to pipe to a built-in command? Will we save output to an instance member for the next command to read?

5. What will one_cmd_plus_hooks run when a command line contains multiple commands? Maybe we can leverage runcmds_plus_hooks() to run the full list of commands from a command line.

[tleonhardt]

I don't have a clear concept of exactly how this feature would/should work. Without one we can't design it. Does anyone else have a clear concept of how they think it should work?

[nvincent-vossloh]

Hello,

I tried (before reading documentation) to do some !cat my_file | my_cmd and realized, after reading documentation that it was not supported and found this issue.

I do not have a clear concept of how it should work, but I can tell you how I would like it to work (I hope both are not exclusive 😉)

Some of the commands I developed ask the user for some data (depending on cli args, or intermediate choices), those datas are either strings, integers, yes/no, that kind of things. They are all validated by a carriage return. The user input is either read using cmd2.Cmd.select() or cmd2.Cmd.read_input.
I wish I could store the user input in a txt file and do a !cat file | my_cmd (or my_cmd < file).

I tried to create a script containing my command followed by the user input and pass it to run_script without success, each line of the file passed to run_script is treated as a command for the shell and not consumed every time select() or read_input() is called.

Please let me know if you need more precision on my use case.

[Hierosme]

Hello,

May be i not understand why the Pipe | require any explanation (i can miss understand).

The pipe is describe from long time ago by the POSIX documentation.
https://pubs.opengroup.org/onlinepubs/9699919799/functions/pipe.html

That is two file descriptor from STDIN, STDOUT, STDERR.

How integrate it to cmd2 for me by modify poutput() for have somewhere a file descriptor to connect to the next piped command as self.stdin.

I can take a look, to code the pipe() function as describe by POSIX, but i'm not sure how integrate inside teh actual code

The doucle || is all ready describre like &&, they use the return code of a command and make logical OR and AND.

[Hierosme]

I have take a look about pipe, but that is all ready exist on os.pipe() and os.pipe2()

https://docs.python.org/3/library/os.html#os.pipe

and

https://docs.python.org/3/library/os.html#os.pipe2 (look to be teh True POSIX pipe)

As describe by POSIX requirements, the os.pipe() should be couple with os.fork(), and the File Descritor should us a POSIX FD.

that is done by it type of code: https://www.tutorialspoint.com/python/os_pipe.htm

#!/usr/bin/python

import os, sys

print "The child will write text to a pipe and "
print "the parent will read the text written by child..."

# file descriptors r, w for reading and writing
r, w = os.pipe() 

processid = os.fork()
if processid:
   # This is the parent process 
   # Closes file descriptor w
   os.close(w)
   r = os.fdopen(r)
   print "Parent reading"
   str = r.read()
   print "text =", str   
   sys.exit(0)
else:
   # This is the child process
   os.close(r)
   w = os.fdopen(w, 'w')
   print "Child writing"
   w.write("Text written by child...")
   w.close()
   print "Child closing"
   sys.exit(0)

A good documentation can be found here: https://epsi-rns.github.io/code/2017/04/17/python-pipe-and-fork.html

[Hierosme]

Here POSIX description of a PIPE:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09

2.9.2 Pipelines
A pipeline is a sequence of one or more commands separated by the control operator '|'. For each command but the last, the shell shall connect the standard output of the command to the standard input of the next command as if by creating a pipe and passing the write end of the pipe as the standard output of the command and the read end of the pipe as the standard input of the next command.

The format for a pipeline is:

[!] command1 [ | command2 ...]
If the pipeline begins with the reserved word ! and command1 is a subshell command, the application shall ensure that the ( operator at the beginning of command1 is separated from the ! by one or more <blank> characters. The behavior of the reserved word ! immediately followed by the ( operator is unspecified.

The standard output of command1 shall be connected to the standard input of command2. The standard input, standard output, or both of a command shall be considered to be assigned by the pipeline before any redirection specified by redirection operators that are part of the command (see Redirection).

If the pipeline is not in the background (see Asynchronous Lists), the shell shall wait for the last command specified in the pipeline to complete, and may also wait for all commands to complete.

[Hierosme]

Here a exemple about oilsshell , that is not a pure POSIX Pipe but that look correct. That method can be use on cmd2 and look more easy to implement.

  # Create private named pipes for this instance.  Anonymous pipes don't work,
  # because they can't be inherited.

  fifo_stdin = os.path.join(THIS_DIR, '_tmp/stdin')
  fifo_stdout = os.path.join(THIS_DIR, '_tmp/stdout')
  fifo_stderr = os.path.join(THIS_DIR, '_tmp/stderr')

  try:
    os.mkfifo(fifo_stdin)
    os.mkfifo(fifo_stdout)
    os.mkfifo(fifo_stderr)
  except OSError as e:
    log('error making fifos: %s', e)

https://github.com/oilshell/shell-protocols/blob/master/coprocess/fcli_invoke.py

---

It look i'm a rare guy concerne by the internal Cmd2 command PIPE. That due to the fact i have start a project arround Cmd2 it consite to create a autonomus, one file binary SHELL. https://gitlab.com/Tuuux/galaxie-shell

It look very strange to have Cmd2 without True PIPE , and no plan to integrate it ...

What is the next step ?

[tleonhardt]

@Hierosme The concerns of the cmd2 developers are around how to implement a pure POSIX pipe within cmd2 and not around how it works in POSIX/UNIX systems.

Supporting a pure pipe syntax where you can pipe from/to cmd2 commands as well as shell commands multiple times would be a great feature, but it would also very significantly complicate our parsing logic as well as how we resolve command aliases/macros.

We would like to implement this feature because we realize it would be a great and natural thing for our users, but we want to make sure we don't unduly burden the maintenance of cmd2 in order to make that happen.

If you have any good ideas for how to implement this in cmd2 without breaking existing features and in a way that would be maintainable, we would absolutely love a PR ;-)

[Hierosme]

Hi @tleonhardt ,

I'm iterested to take a look, and in case send a PR :) or i minima push here my feedback.

Has i understand, how macro/alias (may be sub SHELL command) are implemented will certainly be the pain source .
By experience if i'll touch arround macra/alias it will be with ultra limitated changes, or my PR will never help any one ...

I suppose true POSIX fork will be not allow ... Then i'll try approch with io.StringIO and io.BytesIO and hope add capapility to PIPE huge thing.

We should have something in a "system wait" state, it permit to accept a Keyboard Interruption.

I hope give some good news soon.

Regards

[Hierosme]

I have take a look, and have a better view now.

Actually Internal Command (cmd2.CommandSet) and Sub SHELL command (cmd2.utils.ProcReader) use differente way to deal with return code, stdin and stdout.

The good thing is Sub SHELL command pipe work well, and can permit a workarround with SetupTools and multiple modules/entry point. Easy to interconnect many Command but nothing about the main touble.

During tests i have see Internal command can pipe to a Sub SHELL command the trouble effectivlly about how cmd dedermine if a command is internal or external

I'll continue to look, for now i'll test the workarround with SetupTools

[tleonhardt]

@Hierosme Just curious, did you ever look into this more?

[Hierosme]

Yes i have a working pipe implementation for CMD

https://codeberg.org/Tuuux/galaxie-shell/src/branch/develop/glxshell/lib/ush.py#L160

https://codeberg.org/Tuuux/galaxie-shell/src/commit/c337eac927f24977ea1c97f401b0d14dc3373da4/glxshell/lib/ush.py#L178

https://codeberg.org/Tuuux/galaxie-shell/src/commit/c337eac927f24977ea1c97f401b0d14dc3373da4/glxshell/lib/ush.py#L214

After have succes the pipe on CMD i have take a look on CMD2 but i lake knowledge about CMD2
The pipe is not to mush difficult , implement it on CMD2 is more hard to me.

Any feedback or assistance anout CMD2 integration is welcome

[jpsnyder]

Well this sucks, I was all ready to integrate cmd2 into a project, with grandiose ideas of allowing piping from one command to another in a modularized data manipulation library. Naturally I thought doing something like self.stdin.read() would allow me to pull from piped text/data just like self.stdout.write() would allow you to pipe out.

[kotfu]

There is another option you might consider, which would not require any changes to cmd2, and which may be similar enough to the stdin/stdout thing that you could make it work.

You can create a series of commands in cmd2, which know how to read from and write data to a "shelf" or "container". This "container" could be implemented as an in memory file object like StringIO provided by the io module in the standard library. Instead of creating a pipeline of commands that send data back and forth using stdout/stdin, you give the first command which saves it's output to the container, then run the second command which takes it's input from the container, and saves it's output back to the container. One of your commands could be to save the container to a file. One of your commands could be to load a file into the container. This approach could give you nearly identical functionality to piping, but without requiring us to rip up the internals of cmd2.

[Hierosme]

Hello,

In fact on a True POSIX Pipe only frist and last command work on the true STDIN / STDOUT of the source executor...
Other piped command should use the standard POSIX pipe (implemented by os.dup and os.pipe) .

That really simple, it have Two function

• one for normal (it test if that internal or external command by searching a "do_" command prefix)
• and the other one for piped commands (it evaluate command by command if that internal or external command by use teh previus function)
• The trigger is if the line contain a "|"

The code is really simple i reput here:
Don't care about how i deal with empty line ....

care about it line:
pr = subprocess.run(line.split(" "), env=self.environ)

the sub process run function is POSIX one, then acces a env (where actual os.environ is copy of it parent env). That is how POSIX want deal with child process you have to copy you env to the child env.

That is a big POSIX requirement

    def onecmd(self, line):
        cmd, arg, line = self.parseline(line)
        if not line:
            return self.emptyline()
        if cmd is None:
            return self.default("cmd is None for: %s" % line)
        self.lastcmd = line
        if line == "EOF":
            self.lastcmd = ""
        if cmd == "":
            return self.default("cmd is '' for: %s" % line)
            # return self.default(line)
        else:
            if "|" in line:
                return self.run_multiple_commands(line)
            else:
                return self.run_simple_command(cmd, arg, line)

    def run_multiple_commands(self, line):
        # save stdin and stdout for restoring later on
        s_in, s_out = (0, 0)
        s_in = os.dup(0)
        s_out = os.dup(1)

        # first command takes command from stdin
        fdin = os.dup(s_in)

        # iterate over all the commands that are piped
        for command in line.split("|"):
            # fdin will be stdin if it's the first iteration
            # and the readable end of the pipe if not.
            os.dup2(fdin, 0)
            os.close(fdin)

            # restore stdout if this is the last command
            if command == line.split("|")[-1]:
                fdout = os.dup(s_out)
            else:
                fdin, fdout = os.pipe()

            # redirect stdout to pipe
            os.dup2(fdout, 1)
            os.close(fdout)

            # make tasks it use sys.stdin or/and sys.stdout
            tmp_cmd, tmp_arg, tmp_line = self.parseline(command)
            self.run_simple_command(tmp_cmd, tmp_arg, tmp_line)

        # restore stdout and stdin
        os.dup2(s_in, 0)
        os.dup2(s_out, 1)
        os.close(s_in)
        os.close(s_out)

    def run_simple_command(self, cmd, arg, line):
        if hasattr(self, "do_%s" % cmd):
            func = getattr(self, "do_%s" % cmd)
            return func(arg)
        else:
            pr = subprocess.run(line.split(" "), env=self.environ)
            self.exit_code = pr.returncode

Regards