Add benchmarks

[NickCrews]

Branched off of #965 (comment).

EDIT: See next comment for using ASV instead of @Profile

Place @Profile decorators on bottleneck functions using memory_profiler

List this dependency as extra, so that most users don't need to install it.

Also, to prevent overhead from @Profile getting run always (even when we don't want profiling), wrap it in our own custom decorator that is usually a noop:

def dd_profile(func, *args, **kwargs):
    # Maybe a better way to configure this? Would have to be at import time
    if os.environ["DEDUPE_PROFILE"]:
        # Actually add the profiler wrapper
        return profile(func, *args, **kwargs)
    else:
        # noop
        return func

Next steps are to probably actually make a new branch and apply it to some of the examples?

[NickCrews]

OK, I think a better option is Air Speed Velocity.

• Don't have to modify the codebase at all, benchmarks are separate, like tests
• measure time, memory, and custom attributes (eg we could measure accuracy)
• used by big names like numpy, pandas, scipy
• See examples at that link of static websites, it has built in support for github pages
• Could run on every PR, or could trigger only when we care from a comment like @benchmark main..HEAD --bench benchmarks.TimeSuite.time_range as pandas does

Initially, I would think that we would get a lot of value out of measuring

• time
• peak memory
• accuracy
  on just one or two end-to-end runs (training and evaluating separate). Either could use the test datasets, or could use the examples from dedupe examples

One tricky thing would be to decide how to store the results (eg the metrics collected from past runs). Would want a way to ensure a consistent machine (using github actions? and/or with docker?). Keep the results in this repo, or keep them in a separate repo, possibly using submodules as they mention?

@fgregg @fjsj any thoughts here?

[fgregg]

I love this ides @NickCrews . if you want to try to get something rough and ready set up, i think that would a really good step forward. if it's looking very valuable, then we will figure out the tricky bits.

[NickCrews]

Cool, it might be a bit, but I will try to get to this.

I'll start with the benchmarks in this repo, as those other packages do. Will try to start out without the stored state of past metrics. Will make the decision at game time of which example data to use, haven't explored enough yet to see which I like better.

[NickCrews]

@fgregg @fjsj I'm not well versed in entity resolution, any suggestions on what metrics I should use for "accuracy"? Based on the abstract of https://arxiv.org/abs/1509.04238 (didn't read it yet as I thought maybe you'd have pointers) it sounds like the standard measures like F score and precision might need to get tweaked a little bit. I can also do my own research but if you can tell me where to start it would help. Thanks!

[fgregg]

i think precision and recall are really still the best ones.

[fgregg]

look at canonical.py in tests to see how precision and recall is calculated there.