argparse builtin: Replacement / enhancement for getopts

[andychu]

We need something much better for the Oil language. It's silly that it's hard to write decent command line tools in shell!

• getopts only supports short flags. We want to support long flags, probably using something like Go's flag parsing rules (-abc is one flag, not a shortcut for -a -b -c)
• getopts has a bad API
  • it makes you write too much code. Python's optparse/argparse seems to do the right thing.
  • it has weird implicit variables
  • it has hidden state
• getopts doesn't generate --help for you

More:

• Ideally you should get completions for free by defining opts spec.

Could look something like this:

opts {
  flag -v --verbose Bool "Print more"
  flag -t --timeout Float "Seconds to wait" { default = 1.0 }
  arg src "Source"    # if there's no default, then it's required
  arg dest "Destination"
}

or

var cut_spec = opts() {
  ...
}

That's roughly the API for Python's argparse but in a more compact format.

[drwilly]

What's the rationale for -abc as -a -b -c?

[andychu]

I guess there are three possibilities, all of which are used in some tools. read -da could be:

• the long flag -da, equivalent to --da (Go uses this, and I think Oil will too)
• -d a -- shell builtins allow you to omit the space between the flag and arg. (In this case the delimiter is the value a).
• -d -a -- two different flags without args

I think Oil and Go's way reduces unnecessary variation, which improves readability. You have to type a tiny bit more, which I think is OK.

rm -rf and tar xvf are idiomatic counterexamples, but those can still be invoked that way through Oil of course. You might lose some static analysis features of Oil if you do that though.

[ilyash-b]

In Next Generation Shell, I decided that if main function is defined, NGS will try to match command line arguments to the parameters. It's not finished but already functional.

It is similar to how Perl6 parses command line arguents. The difference is that NGS looks at defined multi-method and tries to parse according to specified parameters as opposed to Perl6 which first parses command line arguments (configurable via %*SUB-MAIN-OPTS) and then tries to match to the main multi-method.

I do think it's very convenient approach and should be considered in case of Oil too.

From script that I wrote today:

F main() {
	# In Perl 6 this message is automatic
	exit("Usage: ${ARGV0} <ENV>")
}

F main(env:Str) {
	AWS::Instance(regions="*", Tags={"env": env}, State="running")...
	...
}

[Artoria2e5]

The sh on Solaris has a nice extension to getopts that allows for describing long args in the optstring. Bascially, to describe a long arg fast for f, you write f(fast). It naturally works with old getopts code, and for older shells you should be able to write a paren-stripping wrapper for it.

As for the -abc thing… I don't like how Go does that, but fine I guess. After all nobody is typing on the old finger-breaking heavy mechanical keyboards anymore. Still, the old (Unix) style shorthand can be made deterministic once an optstring is given, so maybe it's worth a try?

[andychu]

This is very important, but users can also fill it in themselves with the more powerful Oil language, so removing "essential" label

[andychu]

I realized that Hay made some progress on custom commands in a block, like flag arg above:

https://oilshell.zulipchat.com/#narrow/stream/264891-oil-help/topic/getopts.20alternative.3F

[Melkor333]

I guess there are three possibilities, all of which are used in some tools. read -da could be:

* the long flag `-da`, equivalent to `--da` (Go uses this, and I think Oil will too)

* `-d a` -- shell builtins allow you to omit the space between the flag and arg.  (In this case the delimiter is the value `a`).

* `-d -a` -- two different flags without args

I think Oil and Go's way reduces unnecessary variation, which improves readability. You have to type a tiny bit more, which I think is OK.

rm -rf and tar xvf are idiomatic counterexamples, but those can still be invoked that way through Oil of course. You might lose some static analysis features of Oil if you do that though.

I would allow different flavors for -abc, depending on the program, the amount of flags and especially on the usage, one or the other makes more sense.

My suggestions would be an additional command like argstyle letter or argstyle word where letter means that -abc == -a -b -c and word is -abc == --abc (or --abc just doesn't exist in the word style). Having argstyle word as default (plus good documentation) should be enough to point people in that direction.

If we want one right way for parsing parameters we have to be flexible, otherwise some stubborn person will write its own implementation which parses it differently and people will find reasons two parsers aren't enough, etc.

[andychu]

FWIW I removed the Go style flag parsing in Oil awhile ago ! It was not compatible with existing builtins, so we had an awkward split between OSH and Oil builtins

Now all builtins are for OSH AND Oil, which is simpler

So it's possible to have both styles but it will almost certainly lean toward the current Oil / GNU style, not Go style

[andychu]

Some design notes here:

https://oilshell.zulipchat.com/#narrow/stream/325160-oil-discuss-public/topic/Proc.20Flags.2BArgs.20Idea

[andychu]

Summary of Enhancements to Oil

• binding typed args for (myspec, ARGV)
  • Implement binding of typed params to procs #1024
  • (:opts) works in typed args
• ArgSpec :myspec automatically works for Hay blocks
  • Configuration Dialect #951
• define ArgSpec builtin -- capitalized because it's like Hay
• define argparse builtin

That's it? Anything else?

https://oilshell.zulipchat.com/#narrow/stream/325160-oil-discuss-public/topic/Hay.20.3Amyvar.20is.20instantiation.20.3F

[bar-g]

From #297 :

[argparse] is a similar problem [to oil testing framework] in that Oil language features are necessary to support a rich interface
i.e. we can either write arg parsers and test frameworks in Oil, or in the meta-language Python
So that is probably the first thing to work out

I'd like to point out what seems to make a significant difference to me, compared to a testing framwork.

Of course argparse would need to support a rich interface to be able to define fully customized CLIs.

However, hopefully you also realized a modern language should also be able provide a basic CLI interface for any script, out of the box, without needing any definition at all. (In the absence of custom interface definition/disabling.) This can even work for GUIs.

(Useful errors/infos/defaults would even make these features self-discoverable, e.g. myscript --help: "Script does not call any of its procs", "Proc 'mymain' does not provide any docstring to show here" etc.)

Examples, that show this "sane user interface by default" kind of thing:
https://github.com/c-blake/cligen (CLI without explicit argparse definitions)
https://github.com/chriskiehl/Gooey (GUI based on argparse definitions, which can have implicit defaults as of the above)

[andychu]

Hm yeah Raku does something like those

https://stackoverflow.com/questions/29704106/how-do-i-parse-and-validate-command-line-arguments-in-raku-formerly-known-as-pe

I do think the "Python-like parameters vs. shell-like flags" is going to be a design issue, it's really a special case of "procs vs. funcs"

I think my inclination is to keep things more shell-like, or at least have shell-like "fallbacks", but people coming from other languages may have problems with that

more: https://docs.raku.org/language/create-cli

[bar-g]

On the script calling side I'd say providing the shell-like features like "stackable" flags is the right thing for an interactive shell.

Good part of what gives the command prompt it's terse magic are the shorthands, e.g. rm -rf this and tar cvzf this. They make quite a difference when typed interactively a lot, like in admin work and user support. And people coming from other langs likely touch on shell anyway.

[bar-g]

Now I found it, again: On the shell (function) coding side, this could be an example of generating UI even for plain shell vars and functions (OSH): https://github.com/TangoMan75/shoe

Edited:
(Nice direction, could still have some room for some improvements/fixes, e.g. only have (leave it to) auto-completion, instead of auto-generating (allowances) for shorthands of long options, to avoid that these "auto-exposed" shorthand options loose their validity or change meaning in later script versions with the introduction of options.)

[bar-g]

As the shoe example works based on a naming convention, maybe alternatively also think of using a convention to explicitly expose vars/functions to the UI, instead of to make them private/hiden. Which way might be preferable?

[sullyj3]

Support for subcommands with different flags and args would be nice as well.

[bar-g]

subcommands with different flags and args would be nice as well.

Would this be enough:

• global vars are exposed as global flags or options, and function names (oil: proc names), possibly also just those matching some naming convention, are exposed as different "commands" of the script.
• (local) vars of functions (procs) that match the convention are exposed as flags and options to the exposed UI "command" that corresponds to that function's name?

Not sure if, or how, this could be extended to further sub-command levels. sub-functions?

[bar-g]

For example, a naming convention:

arg_* (variables named like this get automatically exposed as options, booleans as a flags)
arg_l_longname="unset" (This variable name defines the option "longname" with one-letter short option "l".)

proc arg_h_hello (same for shell functions and procs names?)

[bar-g]

Turned out that with the typed arg definitions for procs, and especially named-args, proc args can very naturally map to CLI options: #1837

That could very much reduce all necessary configuring to only deviations from the defaults.

(Possibly all in the proc definition and docstring? So defaults do not require having any formal getopts specification.)

[bar-g]

In other words:

Let [proc] option args be defined in such a programmatically and syntactically natural way, that the common external command interface that includes flags and options ("external narrow waist") can also be used as internal interface to calling procs ("internal narrow waist").

Consider parsing flags and options to procs as typed named-args (i.e. optional if defined with a default).
See

• ysh-simple parsing / let command language and typed args compose #1837