Skip to content

How Interactive Shells Work

Jump to bottom Edit New page
andychu edited this page Apr 8, 2020 · 19 revisions

Let's figure out how interactive shells work! bash, zsh, fish, and Oil are significantly different. The goal is to figure out:

  • What interactive features Oil should have. (Right now, it's mostly bash-like)
  • What hooks/APIs it should have so that others can implement nice interactive experiences (like zsh plugins and ble.sh on top of bash)

Please add links or hand-written descriptions in the sections below, or talk to us on #oil-discuss at https://oilshell.zulipchat.com/ (log in with Github)

And if there are too many details, feel free to create new pages and add links to them.

Parts of the Interactive Shell

  • The prompt.
    • In POSIX shell this is $PS1 and $PS2.
    • bash has extensions.
    • Does a shell have to know how wide the prompt is to draw correctly? (note: this depends on unicode chars in the propmt)
  • Shell Autocompletion -- when you hit TAB, what happens
  • Autosuggestions -- does it display suggestions on the prompt line automatically? That is, there is no prompting.
  • Spell correct? zsh has this I think.
  • History. What format is the history stored in?
    • in bash it's a text file, which gives it an odd "two level store" behavior: see issue 324

Research

note: feel free to split this into more wiki pages

How Bash Works

  • It uses GNU Readline. bash and readline are maintained together.
  • It provides $LINES, $COLUMNS, and trap SIGWINCH hooks
  • $PS1 and $PS2 for prompt
  • many options and vars for history, like $HISTFILE, etc.
  • it has a command_not_found hook

How Zsh Works

  • TODO: What are the common plugins and what do they expect from the shell?

How Fish Works

Features

  • autosuggestions

How ble.sh Works

(Related: Running ble.sh With Oil)

1. Processing user inputs

is there a "main loop" ?

ble.sh makes abuse of bind -x which can be used to bind a user-provided command to a user input sequence. ble.sh steals all the user inputs from GNU Readline by binding a shell function to all possible byte values 0-255. The essential idea can be illustrated by the following code (although there are many workarounds for old Bash bugs in actual ble.sh. See lib/init-bind.sh).

declare i
for i in {0..255}; do
  declare keyseq=$(untranslate-keyseq "$i")
  bind -x "\"$keyseq\": process-byte $i"
done

There is no explicit main loop in ble.sh. ble.sh processes received bytes asynchronously one-by-one. In other words, it borrows the main loop of GNU Readline in which Readline calls the shell functions bounded by bind -x. The input byte stream is decoded into the character stream by the specified input encoding (default: UTF-8). The character stream is translated into the key stream by processing special escape sequences that represents cursor keys, function keys, key modifiers, etc. Finally key sequences are constructed from keys in the key stream based on the current keymaps and are dispatched for various operations. All of these input processing is implemented by Bash programs (See src/decode.sh).

Another important Bash feature that ble.sh utilizes is read -t 0 which can be used to test if the next byte in standard input is already available or not. ble.sh uses read -t 0 for polling. For example, ble.sh implements costly operations (e.g. history load, autosuggestions, filtering of menu items, history search) in a kind of coroutines/fibers and perform them in backgrounds while there is no user inputs. When ble.sh detects user inputs by read -t 0, it suspends the fiber and resume it after finishing the processing of the user inputs. Also ble.sh uses read -t 0 to detect the pasting from clipboard (assuming that many inputs in a short time is pasting), etc. (cf the fiber system is implemented by functions ble/util/idle.* in src/util.sh).

API Requirements: To summarize, ble.sh only requires primitive I/O operations receive byte (bind -x) and poll (read -t 0) for its essential part. In other words, Bash/Readline doesn't provide any satisfactory high-level APIs for user-input processing (Bash/Readline provides bind for key bindings but it has tight limitations). If a shell provides some high-level support, a customizable key-binding system and a coroutine system would help users to develop interactive interfaces.

2. Layout and rendering of command line

ble.sh directly constructs the terminal control sequences (escape sequences) by itself. First it determines the graphic attributes (highlighting color, etc.) of each character in the command line (this is another long story, so I'll skip the details). Next, it calculates the width of each Unicode character (it doesn't support combining characters currently) and determine the display position of each character. Then it constructs the control sequences to update the changed part (the characters which has colors or positions different from those in the previous rendering). Finally it outputs the constructed sequences to stderr (See src/canvas.sh for primitive layout/rendering functions, and ble/textarea#* in src/edit.sh for command line rendering).

When ble.sh calculates the layout, it uses the terminal sizes which is available through the special Bash variables LINES and COLUMNS (Of course shopt -s checkwinsize is turned on by ble.sh). Also ble.sh traps SIGWINCH to update the layout and redraw the command line on the size change of terminals. It should also be noted that prompts are also calculated by ble.sh by analyzing PS1 so that ble.sh knows the size and cursor movement of the prompt (See ble-edit/prompt/* in src/edit.sh). When constructing the control sequences, ble.sh also refers to terminfo/termcap by tput command if available (See lib/init-term.sh).

Also, when ble.sh is activated, all the outputs from Bash/Readline should be suppressed. To achive this, ble.sh performs redirection of file descriptors of Bash process using exec >... <....

API Requirements: ble.sh requires a primitive I/O operation output string (printf). In addition, the means to get the current terminal size (LINES and COLUMNS) is needed. The same information can be obtained by external commands such as tput lines and tput cols (ncurses) or resize (xterm utility), yet it is useful to provide them as builtin features (as these commands might not be available in the system). If a shell provides high-level support for this, layout and rendering can be performed by the shell but not by the shell scripts so that the shell scripts only have to specify the characters and their graphic attributes. If the shell provides the prompt calculation, it should also provide the cursor position information after the prompt is printed. The means to suppress/control the I/O of the original shell is also needed.

3. Command execution

how does it execute commands?

ble.sh uses eval. The commands must be executed in the top-level context (i.e., not in the function scope), so ble.sh uses a form of bind -x slightly modified from that described in the above section (1. Processing user inputs):

bind -x "\"$keyseq\": process-byte $i; eval -- \"\$_toplevel_commands\""

Here the shell variable _toplevel_commands is usually empty but contains commands only when some commands should be executed in the top-level context.

Also ble.sh needs to adjust the state of terminals and pty handlers using special terminal sequences and also the external command stty before and after the command execution. Those adjustments are also included in _toplevel_commands

API Requirements: The ble.sh requires a means to execute commands in the top-level context (direct eval in bind -x). Also ble.sh uses the external command stty to adjust the pty handler state which might be better to be built in the shell.

4. Summary

What does it expect from bash?

  • bind -x (read), read -t 0 (select/poll), printf (write), exec redirection (dup2, fcntl, etc.), eval
  • $LINES, $COLUMNS
  • it needs to trap SIGWINCH (todo: test this signal in Oil)

How Oil Works

Oil is mostly bash-like now. I added a zsh-like completion interface that doesn't scroll every time you hit TAB, but it has some bugs, like #257.

Source Code

  • zsh-$VERSION/Src/Zle/zle_* -- 20K lines of C code
  • GNU readline: ~38K lines of C code
  • ble.sh: 30K+ lines of bash code
  • yash: has its own line editor that does what we want! It doesn't scroll the prompt!
    • but it doesn't support Ctrl-R, etc.

Drawing Models

  • vim/emacs: draw on the "alternate screen" ?
  • bash: this is just a normal app
  • fish/zsh/ble.sh? Some kind of hybrid
    • ble.sh predicts the location though. I'm not sure if fish/zsh do though. I guess fish must.
Add a custom footer
Add a custom sidebar
Clone this wiki locally