Comment handling

[AnnieAtHasura]

Proposed enhancement:

Add a repeatable option comment(Start, End) that discards all material between start and end.
One could add an option comments that causes the tokenizer to include comment(Text) as a token, where Text is the contents of the comment.

So, to get Prolog comments, comment('%', '\n') or to get C style comments, comment('/*', '*/')

Issue: What about nested comments?

[itampe]

I did some prottyping of a comment ideom. So there are now two main predicates,

comment(S,E),
comment_token(S,E,Text)
comment_token_rec(S,E,Text)

comment_rec(S,E), /* not yet */

The text version will collect the text between and included S,E with the interpretation that
S(HS) will make match the start and put a list representing the tag in HS ens similar for the end tag E for the non text version, S will just match and not produce any representation. The rec versions will make sure that recursive comments works just fine. If there is a missing end of comment the system should insert the missing end of comment, perhaps we should signal an error.

I did some prototyping here, check it out if you like.
comment.txt

[shonfeder]

Two questions and one request:

1. Does "recursive comments" mean the same as "nested comments"?
2. Why do we need to use append/3 in the DCG rules? Shouldn't we be able to handle these with dcg rules?

Any reason not put this into a PR (along with some tests to ensure it's working as expected)? This will let us have inline discussions about particular points of the code. The tests will also help show us some examples, which would help me nail down what's going on (my DCG abilities are rusty :) ).

[itampe]

1. yes nested and recursive is synonymous to me.
2. My dcg foo is not good as well. But if one produces a new stream out of the actions then yes
   we should be able to handle this in the dcg's directly. things inside {} should be normal prolog code so
   I was produceing the output stream via the arguments and there append is needed.

Also I will try to put it into a PR

[Anniepoo]

I have approved #33 (review)
and merged. Current solution is not integrated. Once we decide how options get handled, I'll integrate and we can review code at that time.