Structured Data in Oil

Oil will parse text so you don't have to!

Links / Use Cases

• What is a Data Frame? (In Python, R, and SQL) (blog)
• Git Log in HTML (blog)
  • After that I wrote Structured Data Over Pipes
  • And Unix Tools
• Oil and the R Language
• Other Oil Use Cases:
  • release.sh generates releases.html
  • benchmarks/*.{sh,R} generates osh-parser report, etc.
• lobste.rs comment on Oil philosophy for structured data

Design Docs

• Done: QSN: A Familiar String Interchange Format
• TSV2 Proposal (renamed QTSV)
• each Keyword in Oil -- augments xargs, uses TSV2
• Explicit Framing Protocol Proposal

Curiosities

• Shell Almost Has a JSON Analogue

Projects

All of these are obviously supported because Oil is a shell! But there are advantages to a built-in expression language. (It's deferred for 2020.)

• CSV / TSV / etc.

  • https://github.com/johnkerl/miller -- Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
  • https://csvkit.readthedocs.io/en/1.0.3/
  • https://github.com/BurntSushi/xsv -- A fast CSV command line toolkit written in Rust
  • https://github.com/sustrik/uxy

• JSON

  • https://github.com/benbernard/RecordStream -- commandline tools for slicing and dicing JSON records
  • https://github.com/aanastasiou/pyjunix
  • http://jsonlines.org/ and http://ndjson.org/ -- what's the difference?
  • https://github.com/kellyjonbrazil/jc from https://news.ycombinator.com/item?id=22366638
  • https://github.com/tomnomnom/gron -- convert JSON to grepable and sedable form, and back.

• HTML

  • https://lobste.rs/s/m4nhpb/danburzo_hred_reduce_html_json_from

• Other Text

  • https://www.gnu.org/software/recutils/

• Binary

  • https://relational-pipes.globalcode.info/v_0/index.xhtml -- binary format

• Common Output Format for unix-like tools: https://github.com/aniou/cof/wiki/Draft (draft rather than code)

  • netstat example, etc.

Examples

Miller:

$ mlr --icsv --opprint --barred \
  put '$tiv_delta = $tiv_2012 - $tiv_2011; unset $tiv_2011, $tiv_2012' \
  then sort -nr tiv_delta flins.csv 

It has a nice expression language. You compute new columns as a function of older ones, then drop them.