Oil should have a slurp builtin

[andychu]

Problems:

• https://mywiki.wooledge.org/BashFAQ/105 -- reading a file without a newline can unexpectedly fail with status 1 in some contexts. Because no delimiter is read.
• @david-a-wheeler pointed out that it's sometimes undesirable that $(cat myfile.txt) omits the newline.

So slurp can be used as follows:

slurp myvar < myfile.txt  # includes newlines if any, status is 0

grep foo *.c | slurp myvar    # works because of shopt -s lastpipe

---

related:

• https://metacpan.org/pod/Perl6::Slurp
• we already have getline, because read -r isn't the default.

[Crestwave]

So like:

while read -r line; do
    myvar+=$line$'\n'
done <myfile.txt
[[ -n $line ]] && myvar+=$line

But shorter and probably much faster as well, of course.

[akinomyoga]

Another solution with mapfile (which preserves the delimiter unless -t is specified):

mapfile myvar < myfile.txt
IFS= eval 'myvar="${myvar[*]}"'

If one can assume that there is no NUL byte (which cannot be handled properly in Bash) in the file, here is the implementation in ble.sh (ble/util/readfile). You can just write in a single line:

mapfile -d '' myvar < myfile.txt

Or, with read,

IFS= TMOUT= read -r -d '' myvar < myfile.txt || [[ $myvar ]]

[andychu]

Ah OK mapfile -d '' myvar is pretty simple.

And Oil supports NUL bytes, so that would work with either mapfile or slurp.

So I guess slurp is not strictly necessary then, although it's shorter. I suppose we can implement it as a "polyfill" shell function, like

• provide 'checked' wrapper for grep, test, etc. #474 -- testing the exit code of grep -- success/failure vs. true/false
• shortcut for $(dirname $(readlink -f $0)) #587 -- find the directory of the current script (respecting symlinks)

[david-a-wheeler]

It might be easier to simply add another option or two to "read"; read is already (essentially) reserved. E.g., "-R" (raw) option would ignore IFS & not re-interpret anything, and simply load stdin's contents into the named variable (up to the byte length limits if given).

[andychu]

Yeah that is possible -- it's a matter of style. If that were the case I'd want to implement --long flags ( #465 ) first (at least the flag parser support).

Oil currently has:

• getline, which is roughly like read -r, and could be read --line. Although somehow that seems long and awkward. Also read is unbuffered, but getline is buffered I/O which I think is also useful.
• write to avoid the -- problem with echo

So it could be read --all instead of slurp. But I don't like that the common things require flags, while the uncommon thing doesn't. So that's why I wanted to "promote" these more common behaviors to their own builtins.

I added that principle here:

https://github.com/oilshell/oil/wiki/Language-Design-Principles

e.g. I believe read -r should have been the default, and is read is a bit error prone as is. And I think read -r vs read -R is too confusing without long flags.

Thanks for the feedback.

[andychu]

It is possibly safer to create read --line, and read --all, and then optional "aliases" getline and slurp.

But shells already stomp all over other commands like time, kill, printf, etc. And there is a mechanism to diable that -- command time, command slurp, etc.

[andychu]

This is related to #588, which is basically namespace management for the "first word".

And to some degree #453 for function namespaces.

[andychu]

OK we now have

read --line :myline   # optional : for variable names
echo "$myline"

read --all :x
echo "$x"

I have been getting feedback that it's better not to stomp on the "first word" namespace, so these are indeed long flags, rather than slurp and getline.

Feedback is welcome!

[andychu]

https://www.oilshell.org/release/0.8.3/