Add EOF token to grammar

[Lotes]

Adds the ability to explicitly parse the EOF token from Chevrotain.

[Lotes]

@Yokozuna59 I had to discard the other PR. The current implementation only allows EOF within parser rules - so no terminals or terminal fragments.

I am wondering if it is still helpful. When I was searching for test examples, this EOF feature seemed not to be very powerful.

[Yokozuna59]

The current implementation only allows EOF within parser rules - so no terminals or terminal fragments.

I am wondering if it is still helpful. When I was searching for test examples, this EOF feature seemed not to be very powerful.

@Lotes I think the main usage in my mind about it is to use it as EOL rule as @msujew said:

https://github.com/mermaid-js/mermaid/blob/6e0f41180fcb384df2188b77c93e8620a38d2e5d/packages/mermaid/src/diagrams/pie/parser/pie.jison#L84-L88

So it should be used as fragment rule:

fragment EOL:
  NEWLINE+ | ';' | EOF
;

The main issue we're facing is that each attribute should be in a separate line, so we can't use \s+ as hidden rule. But the last line mayn't end with \n, that why we override Lexer to add one:

https://github.com/mermaid-js/mermaid/blob/205360c1093ad0059989e05d5efecc4f019225b7/packages/parser/src/language/common/commonLexer.ts#L6

[Lotes]

I think in your context that you have described, the new EOF keyword will help, so you are not dependent on the lexer hack. Since EOL is a parser rule, it will work.

[msujew]

Looking pretty good, but there's a test case I'm missing (which is the main reason to use EOF at all): Can you create a test that builds a line based language and confirm that it parses as expected? I.e. something like this:

Lines: lines+=Line*;
Line: text=ID (EOL | EOF);
terminal EOL: /\r?\n/;

[Lotes]

@msujew New tests were pushed and branch was rebased. Wingardium Approviosa? :)

[msujew]

Alright looks good. Can you please remove the changes to the package-lock.json before merging?