Cmd2 loses lines of history on Windows when use_rawinput=True

[nfnfgo]

Version Info

Here is my version info.

• python: 3.12.4
• cmd2: 2.4.3
• windows: Windows 11
• powershell: 7.4.5

Issue

I'm using cmd2 to create an interative CLI on Windows. And as the title implies, It seems there is terminal output issue when use_rawinput=True.

Concretely, if the output lines count of cmd2 application is over the vertical size of the Terminal on Windows, the overflowed line will directly disappeared from the terminal console history. A video is attached below to demostrate the issue:

The word "history" in title means the history of the content being displayed on terminal, not the history of input commands.

cmd2_on_win_terminal.mp4

The video is using Terminal application on Windows, but the integrated terminal in VSCode could also reproduce this issue on my computer. The code used is also attached below:

Code used in the video

import sys
import os

from typing import Optional, List, Iterable

import cmd2
from cmd2 import Settable, Statement, utils
from cmd2 import (
    Cmd2ArgumentParser,
    with_argparser,
    with_argument_list,
    with_default_category,
)


class ExampleCmd2Application(cmd2.Cmd):
    def __init__(self):
        self.use_rawinput = True
        super().__init__(startup_script=".sudokurc", silence_startup_script=True)
        self.count = 0

    def do_iter(self, *args, **kwargs):
        self.poutput(self.count)
        self.count += 1


def main():
    app = ExampleCmd2Application()
    app.cmdloop()


if __name__ == "__main__":
    main()

This issue disappeared once I change the code above into:

    def __init__(self):
-       self.use_rawinput = True
+       self.use_rawinput = False
        super().__init__(startup_script=".sudokurc", silence_startup_script=True)
        self.count = 0

---

After some simple investigation, it seems that this issue is related to either readline, Windows Powershell or Windows Terminal, following are some relevant links:

microsoft/terminal#10975 (comment)
PowerShell/PSReadLine#724

Based on the search result and the fact that I failed to reproduce this issue in GitHub Codespace in Linux environments, I assumed this is a platform-specific issue which only exists on Windows.

[kmvanbrunt]

I'm on Windows 10 and I've tested with the built-in Powershell (5.1.19041.5007) and I installed Powershell 7 (7.4.5).
Unfortunately, I can't reproduce the behavior you're seeing.

Can you run pip freeze to check what version of pyreadline3 is installed? I am using 3.5.4, which is the newest version.

[nfnfgo]

Thanks for the reply!

I was using pyreadline3==3.4.1 while recording the demo. However, the issue still persists even after upgrading to pyreadline3==3.5.4.

I'm managing my environment with conda, but I'm not sure if the problem is related to using conda in the terminal.

I also tried uninstalling and reinstalling the latest version of cmd2, but that didn’t fix it.

You mentioned you're using Windows 10, and I'm not sure if you're working with the new Windows Terminal or the older Command Prompt. If you're using the older Command Prompt without any issues, this might be a Windows 11 or Windows Terminal-specific issue.

[nfnfgo]

It seems that the demo works with no problem if I don't use the new Windows Terminal applications

[nfnfgo]

After some debugging, I think this issue is causes by the usage of package pyreadline3. Here is my debugging process.

Debugging

First of all, this issue only occurred when use_rawinput=True, so I found this in cmd2:

cmd2/cmd2/cmd2.py

Lines 3231 to 3246 in 3062aaa

	if self.use_rawinput:
	if sys.stdin.isatty():
	try:
	# Deal with the vagaries of readline and ANSI escape codes
	escaped_prompt = rl_escape_prompt(prompt)
	with self.sigint_protection:
	configure_readline()
	line = input(escaped_prompt)
	finally:
	with self.sigint_protection:
	restore_readline()
	else:
	line = input()
	if self.echo:
	sys.stdout.write(f'{prompt}{line}\n')

At first I suspected the issue is somehow caused by configure_readline() and restore_readline(), so I try delete these two function calls, however that didn't work.

Then I started to check this input line:

cmd2/cmd2/cmd2.py

Line 3239 in 3062aaa

line = input(escaped_prompt)

I used the debugger to step into the execution of this input() function, and it went into the pyreadline3 library, below is the call stack.

_update_line (<pkgs_path>\pyreadline3\rlmain.py:535)
readline_setup (<pkgs_path>\pyreadline3\rlmain.py:602)
readline (<pkgs_path>\pyreadline3\rlmain.py:605)
hook_wrapper_23 (<pkgs_path>\pyreadline3\console\console.py:842)
read_input (<pkgs_path>\cmd2\cmd2.py:3006)
_read_command_line (<pkgs_path>\cmd2\cmd2.py:3053)
_cmdloop (<pkgs_path>\cmd2\cmd2.py:3136)
cmdloop (<pkgs_path>\cmd2\cmd2.py:5285)
main (<cwd>\main.py:105)

---

The input() function triggered the hook_wrapper_23 in pyreadline3:

Full code of hook_wrapper_23

def hook_wrapper_23(stdin, stdout, prompt):
    """Wrap a Python readline so it behaves like GNU readline."""
    try:
        # call the Python hook
        res = ensure_str(readline_hook(prompt))  # <----------------- here
        # make sure it returned the right sort of thing
        if res and not isinstance(res, bytes):
            raise TypeError("readline must return a string.")
    except KeyboardInterrupt:
        # GNU readline returns 0 on keyboard interrupt
        return 0
    except EOFError:
        # It returns an empty string on EOF
        res = ensure_str("")
    except BaseException:
        print("Readline internal error", file=sys.stderr)
        traceback.print_exc()
        res = ensure_str("\n")
    # we have to make a copy because the caller expects to free the result
    n = len(res)
    p = Console.PyMem_Malloc(n + 1)
    _strncpy(cast(p, c_char_p), res, n + 1)
    return p

The hooks above triggered readline_hook function. (At this point the readline_hook is a pyreadline3.rlmain.Readline object)

Then we reached Readline.readline_setup(), and this setup function calls two functions:

def readline_setup(self, prompt=""):
    BaseReadline.readline_setup(self, prompt)
    self._print_prompt()
    self._update_line()

It seems that the _update_line() is in charge of updating line and control the scroll behaviour when a new line has been entered. And I went into _update_line():

    def _update_line(self):
        c = self.console
        # ...
        if (y >= h - 1) or (n > 0):
            c.scroll_window(-1)             # <--------
            c.scroll((0, 0, w, h), 0, -1)   # <--------
            n += 1

Here two scroll-related functions has been called. Then I checked the scroll() function:

    def scroll(self, rect, dx, dy, attr=None, fill=" "):
        """Scroll a rectangle."""
        if attr is None:
            attr = self.attr
        x0, y0, x1, y1 = rect
        source = SMALL_RECT(x0, y0, x1 - 1, y1 - 1)
        dest = self.fixcoord(x0 + dx, y0 + dy)
        style = CHAR_INFO()
        style.Char.AsciiChar = ensure_str(fill[0])
        style.Attributes = attr
        
        return self.ScrollConsoleScreenBufferW(  # <------- Here
            self.hout, byref(source), byref(source), dest, byref(style)
        )

As the docstring suggests, """Scroll a rectangle.""", this function seems to handle scrolling behavior within a defined rectangle, which aligns with the issue I'm facing.

I looked into the console function used in scroll(), ScrollConsoleScreenBufferW, and found relevant information in the Microsoft Console API documentation.

The documentation describes the behavior of this API as follows:

Moves a block of data in a screen buffer. The effects of the move can be limited by specifying a clipping rectangle, so the contents of the console screen buffer outside the clipping rectangle are unchanged.

I believe this API is causing the issue. Additionally, the PSReadLine GitHub PR seems also removed the usage of ScrollConsoleScreenBuffer to resolve a similar scrolling issue.

The one used in pyreadline3 has an extra alphabet W, and I'm not sure if it's related to the ScrollConsoleScreenBuffer in the Microsoft API docs.

Workarounds

After having all the info above, I tried to replace the scroll() and scroll_window() with console.write('\n') and this time the scroll behaviour becomes the one I want:

# pyreadline3/rlmain.py
class Readline(BaseReadline):
    # ...
    def _update_line()
        # ...
        if (y >= h - 1) or (n > 0):
	        # replace scroll and scroll_window with a single console.write('\n')
            c.write('\n')
            # c.scroll_window(-1)
            # c.scroll((0, 0, w, h), 0, -1)
            n += 1

Also, the comment in pyreadline3 says that one extra line is preserved for IEM statusbar. And I guess this is the reason why there is one extra empty line at the bottom when set use_rawinput=True in cmd2:

• use_rawinput = True

• use_rawinput = False

If we just let the program ignore this (change the if condition y >= h - 1 to y >= h), those two scroll functions will not be triggered and the issue is also resolved.

# pyreadline3/rlmain.py
class Readline(BaseReadline):
    # ...
    def _update_line(self):
        c = self.console
        # ...
        x, y = c.pos()  # Preserve one line for Asian IME(Input Method Editor) statusbar
        w, h = c.size()
        # if (y >= h - 1) or (n > 0):
        if (y >= h) or (n > 0):
            # after changing the condition, the demo python program 
            # will never reach inside this if block, thus no more 
            # scrolling issue.
            c.scroll_window(-1)
            c.scroll((0, 0, w, h), 0, -1)
            n += 1

These are two possible workarounds for my specific case. However, I am currently unable to find the approach to resolve this issue.

[tleonhardt]

One possibility we might be able to explore would be to detect if the cmd2 application is running in Windows Terminal and if so not use pyreadline3.

My understanding is that the new Windows Terminal is pretty much a POSIX-compliant terminal complete with support for things like ANSI escape codes and the like. I don't have Windows to experiment, but I think there is a good chance stuff would "just work".

@kmvanbrunt Based on all of the data provided by @nfnfgo do you have any smart ideas?

[kmvanbrunt]

Not using pyreadline3 means no tab completion. As annoying as losing scrollback lines is, not having tab completion in Windows Terminal is worse.

We should just document the limitations of running cmd2 on Windows due to current bugs in pyreadline3.

Some that come to mind are:

1. Limited to first 8 terminal colors for the prompt.
   BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, LIGHT_GRAY
2. Reverse-search issues documented here Reverse Search History Not Functioning Correctly on Windows #1315.
3. Losing scrollback lines in Windows Terminal.