Add loss value metric based on optimal performance definition

[jteijema]

A loss metric value based on the distance between the perfect recall curve and the actual recall curve.

[J535D165]

Failing tests seem unrelated, but do you have an idea?

[J535D165]

Linter fixed in main branch.

[jteijema]

No, but I figured it wasn't on my branch indeed. Ill take another look!

[jteijema]

The other tests are still failing. I think tf-keras needs to be a req for ASReview. But more importantly, why is it importing transformers when running our tests...

[jteijema]

Tests are failing because of something fixed in #67 . Consider merging 67 and merging main into this branch to get the tests functional again.

[J535D165]

Oke, remove the draft status of that pr.

[J535D165]

This PR needs a test and documentation. I'm not convinced this loss metric computes what's described in the PR description.

The following tests both fail:

def test_metric_loss_best():
    labels_best = [1, 1, 1, 0]
    loss = _loss(labels_best)

    assert_almost_equal(loss, 0)

def test_metric_loss_worst():
    labels_worst = [0, 0, 1, 1]
    loss = _loss(labels_worst)

    assert_almost_equal(loss, 1)

[jteijema]

This PR needs a test and documentation. I'm not convinced this loss metric computes what's described in the PR description.

The following tests both fail:

def test_metric_loss_best():
    labels_best = [1, 1, 1, 0]
    loss = _loss(labels_best)

    assert_almost_equal(loss, 0)

def test_metric_loss_worst():
    labels_worst = [0, 0, 1, 1]
    loss = _loss(labels_worst)

    assert_almost_equal(loss, 1)

Yeah you're right. the calculation was okay, but it wasn't normalized. Here's the new approach:

[J535D165]

Thanks for the update. Looks like a good step forward.

I'm also interesting in adding this metric to asreview/asreview. This can be nice for the CLI simulation interface. Are you interested in contributing a copy there?

[jteijema]

[J535D165]

Thanks for this contribution to the performance metrics!