-
-
Notifications
You must be signed in to change notification settings - Fork 162
Oil Parser Generator Project
Back to Tasks Under NLNet Grant
This is an introduction to an important subproject of https://www.oilshell.org/
Oil is developed "middle out", with an "executable spec" in Python, which is then semi-automatically translated to C++.
Much of the code works in C++, but the expression parser does not. It needs special handling.
In Python, the oil_lang/grammar_gen.py tool reads the grammar oil_lang/grammar.pgen2. Then it spits out a bunch of parse tables in Python's "marshal" format. At runtime, the pgen2/ library reads it.
So instead of outputting Python data structures in "marshal" format, we want to output C data structures just like Python itself does it. (before Python 3.8, when they switched to PEG.)
- Issue 594: Generate Parse Tables for pgen-native, and integrate it into oil-native. This is part of the translation to C++. Right now we only have a slow parser in Python for the Oil expression language.
How to Parse Shell Like a Programming Language explains our parsing approach. This already works in Python:
$ bin/oil --ast-format text -n -c 'echo "hello $name"'
(command.Simple
words: [
(compound_word parts:[(Token id:Id.Lit_Chars span_id:0 val:echo)])
...
And it's already translated to C++:
$ _bin/cxx-dbg/osh_eval -n -c 'echo "hello $name"'
(command.Simple
words: [
(compound_word parts:[(Token id:Id.Lit_Chars span_id:0 val:echo)])
...
This part does not use pgen2.
~/git/oilshell/oil/Python-2.7.13$ head -n 15 Python/graminit.c
/* Generated by Parser/pgen */
#include "pgenheaders.h"
#include "grammar.h"
PyAPI_DATA(grammar) _PyParser_Grammar;
static arc arcs_0_0[3] = {
{2, 1},
{3, 1},
{4, 2},
};
static arc arcs_0_1[1] = {
{0, 1},
};
static arc arcs_0_2[1] = {
{2, 1},
oil_lang/grammar_gen.pyoil_lang/grammar.pgen2-
_devbuild/gen/grammar.marshaland_devbuild/gen/grammar_nt.py(non-terminals) -
oil_lang/expr_parse.py-- a wrapper for the generated parser - The
pgen2/directory- parse.py and more
-
pgen-native/dir -- this is just a copy of Python, imported by a contributor