Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure Parser only reads tokens as necessary #135

Open
faultyserver opened this issue Jan 29, 2018 · 0 comments
Open

Ensure Parser only reads tokens as necessary #135

faultyserver opened this issue Jan 29, 2018 · 0 comments
Labels
semantics.parser Any issue relating to changing the semantics of parser code. semantics Generic label for semantics issues. All semantics issues should have this tag.

Comments

@faultyserver
Copy link
Member

One of the original goals of the design of the Parser was to make it able to work with streamed input, where the entirety of the input is not known immediately. The implementation around this says that every parsing method should only consume the tokens that it requires.

Something that came up through some initial work on implementing a REPL for Myst was that, in some cases, the parser will read one or two tokens ahead to determine which branch of the parser to descend into. This caused an issue where the REPL would ask the user for more input even when the expression they entered was complete.

A different possible solution for this is to somehow pad the input that the REPL passes to the parser so that any look-ahead is unambiguously dealt with. With the use-case of a REPL, padding the input with newlines should be enough to deal with this, as part of the grammar says that newlines act as expression delimiters, so the presence of a newline after a complete expression is enough to indicate the end of the expression (e.g., no methods are chained after the expression).

@faultyserver faultyserver added semantics Generic label for semantics issues. All semantics issues should have this tag. semantics.parser Any issue relating to changing the semantics of parser code. labels Jan 29, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semantics.parser Any issue relating to changing the semantics of parser code. semantics Generic label for semantics issues. All semantics issues should have this tag.
Projects
None yet
Development

No branches or pull requests

1 participant