Unix Tools

The shell has interacts with a set of Unix tools in /bin and so forth. However, in many cases, those tools have grown functionality that overlaps with shell.

Unix Tools ...

That Start Processes (in parallel)

• make and other build tools. make -j for parallel builds.
• xargs, -P for parallel execution, -I {} for substitution
  • Also GNU Parallel, which is mentioned in the bash manual
• find -exec and -exec +

That Have Expression Languages

Expression languages must be fully recursive to count here.

With no lexer:

• find -- -a -o ! ( )
• test -- -a -o ! ( )
• expr -- arithmetic, subsumed by $(())

Languages with lexers:

• awk
• dtrace -- modelled after awk.

Honorable mention:

• strace also has a little expression language, but it's not fully recursive

That Use Regexes

• grep, grep -E
• sed, sed --regexp-extended in GNU sed
• awk (extended only)
• expr
• find -regex
• bash itself

That Receive Code Snippets (Remote Evaluation)

• tar has a --sed option

That Have Printf-Style Formatting

See Appendix A: How to Quickly and Correctly* Generate a Git Log in HTML

• find -printf (arbitrary filenames)
• stat -c (arbitrary filenames)
• curl --write-out %{response_code} -- URLs can't have arbitrary characters?
• printf itself (coreutils)
• time (/usr/bin/time) -- mostly numbers
• date -- mostly numbers
• bash
  • the printf builtin
  • the time builtin and the TIMEFORMAT string -- mostly numbers
  • the prompt string: \h \W
• ps --format

Non-standard tools:

• git log --pretty=format: (arbitrary descriptions)
• hg log --template -- http://hgbook.red-bean.com/read/customizing-the-output-of-mercurial.html (doesn't have \0 as far as I can tell.) Mercurial has its own template language like date: {date|isodate}\n\n (no $).

NOTE: grep should have a syntax for captures, like $1 $2 name: $name age: $age. sed just has & for the matched group.

With Single Field Substitution

These should be replaced with $_ or @_ ("it").

• xargs -I {} -- echo {}
• find -exec {} +

With Tabular Output

• find / ls
• ps
• df (has -h and -H human-readable option, --output[=FIELD_LIST] but no format string)
• du -- has -0 for NUL output
• TODO: look at netstat, iostat, lsof, etc. Brendan Gregg's pages.

With File System Path Matching

• du --exclude
• rsync --include --exclude
• find -name, -regex, -wholename, etc.

Misc Expression Languages

• getopts builtin spec, and /usr/bin/getopt
  • leading : means to do different error handling! Instead of the arg. Gah.

The Worst Offender

find starts processes (in parallel), it a recursive boolean expression language, it has regexes (and globs), and it has field substitution. It should be part of the shell!

It also doesn't give good parse error messages. Sometimes it just says "find: invalid expression" with no location information.

Wow this is crazy too:

$ find -regextype -help
find: Unknown regular expression type ‘-help’; valid types are ‘findutils-default’, ‘awk’, ‘egrep’, ‘ed’, ‘emacs’, ‘gnu-awk’, ‘grep’, ‘posix-awk’, ‘posix-basic’, ‘posix-egrep’, ‘posix-extended’, ‘posix-minimal-basic’, ‘sed’.

I didn't know there were that many regex types!

Families of Unix Tools

• CSV, JSON, HTML, XML, recfile, 1991 paper!
  • https://www.reddit.com/r/oilshell/comments/719p6c/how_to_quickly_and_correctly_generate_a_git_log/dndp9uq/