How to track position?

[fponticelli]

Hi, thanks for the great library. How would you go about keeping track of the current position? Let's say I want to design a language that has some syntax checking that needs to happen after everything has been parsed, I still want to be able to report syntax errors at the right position from the source.

[teo-tsirpanis]

Hello, thanks for the question and thanks for using Farkle! I am glad to see such appreciation for it.

That's a good question. Farkle actually has the ability to allow user code to track the position of tokens. Let's take a look at an example from the quick start guide:

let numberStringRegex =
    Regex.regexString @"\d+(\.\d+)?(e[+-]?\d+)?"
    |> terminal "Number" (T(fun _ x -> float (x.ToString())))

See the first ignored parameter in the last line's lambda? It holds an object of type ITransformerContext, which has properties that expose the token's starting and ending position. You can write with that something like the following:

let numberStringRegex =
    Regex.regexString @"\d+(\.\d+)?(e[+-]?\d+)?"
    |> terminal "Number" (T(fun ctx x ->
        printfn "Number at %O" ctx.StartPosition
        float (x.ToString())))

Let me know if everything went well. 🙂

[teo-tsirpanis]

I also updated the quick start guide. It explains a bit more how can the ITransformerContext type be used.

[fponticelli]

Thank you very much and it seems exactly what I needed. I'll try it this evening ;) ... it might be interesting to add an optional piece of metadata there to keep a reference to a source file. Is that what the ObjectStore is for?

[teo-tsirpanis]

The ObjectStore is used to share state between multiple transformers. It could be used for example by a compiler to write an easily accessible symbol table. I haven't needed to use it, but I added it because FsLex has a similar feature.

Adding metadata about the source file's name is something I have been thinking and will be added in the future. Until then, I have thought of two workarounds:

1. Include the file name when catching an error. If something goes wrong with parsing, you have an opportunity to state on which file it happened like this:

   let parseFile fileName =
       RuntimeFarkle.parseFile myRuntimeFarkle fileName
       |> Result.mapError (fun err -> fileName, err)

   The function above returns a Result<MyType, string * FarkleError>, where the error's first tuple part is the file name in which the error occured.

2. Directly add the file name in the object store. This is a bit more involved but if, for some reason your transformers need the source file's name, you can add it by manually creating a Farkle.IO.CharStream, which is otherwise automatically managed by the RuntimeFarkle.parseFile function. Here is some sample code:

   open Farkle
   open Farkle.IO
   open System.IO
   
   // Both the transformers and your code invoking Farkle's parser will have to access a common key.
   let sourceFileNameKey = "SOURCE_FILE"
   
   // This is an example of how to use it.
   let exampleTerminal =
       Regex.string "test"
       |> terminal "Test Terminal" (T(fun ctx data ->
           // If you are parsing a literal string instead of a file, this key won't exist in the object store.
           let sourceFileName = ctx.ObjectStore.[sourceFileNameKey] :?> string
           printfn "Found %s on file %s at position %O" (data.ToString()) sourceFileName ctx.StartPosition))
   
   // [...]
   
   let parseFile fileName =
       use tr = File.OpenText(fileName)
       use cs = new CharStream(tr)
       cs.ObjectStore.Add(sourceFileNameKey, fileName)
       RuntimeFarkle.parseChars myRuntimeFarkle cs

[fponticelli]

Great, thanks!

[segphault]

@teo-tsirpanis is there an equivalent way to get the start/end positions for a non-terminal?

[teo-tsirpanis]

Unfortunately no, you would have to track the positions of all terminals, and the starting/ending position of a nonterminal becomes the starting/ending position of the first/last member of the nonterminal's production.

It would be pretty expensive for Farkle to track these positions by itself.