Benchmarking

[snadi]

Initially, I thought that we cannot do precision with our benchmark since there may still be useful sentences we don't know about. Thus, a sentence we mark as false positive since it doesn't exist in our benchmark may still be a "good" sentence.

However, I realize this is actually not true. We looked at all if sentences in the threads we analyzed. Thus, we can calculate both precision and recall accurately. We cannot say those are the only useful sentences, but we can say those are the only useful if sentences. Thus, the low numbers we are getting means we were not doing that well in the first place.

I need to look at this a bit more and see what we did while in DysDoc https://docs.google.com/spreadsheets/d/17_76WimA6MyQ2LSVNdHXHIzFaubTeDMhpFOXI3u_wBw/edit#gid=0

[snadi]

As a test, add heuristic 0 which is any if sentence. This should give perfect recall and precision