Support stream-based tokenization

[shonfeder]

For large inputs we want to be able to process one line at a time, so we don't have to read the entire thing in to memory.

[Anniepoo]

Use phrase_from_file. This reads ahead only enough to perform the reduce.
If you don't leave choice points when you emit a token it should neatly dispose of list.
Also, it works with lazy_list_location and so on for neatly reporting errors.
The only time it would consume a lot of memory would be eg an unclosed string if you allow newlines in files, or trying to read a file by lines, and a line by line mode breaks under same conditions.

I would advocate for not doing this, it hard codes line breaks are token breaks into the tokenizer in a way you may come to regret. EG if you attempt to tokenize SWI-Prolog code, you have to deal with multiline strings.