Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics overcount. #11

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Metrics overcount. #11

wants to merge 2 commits into from

Conversation

tabahi
Copy link

@tabahi tabahi commented Oct 20, 2024

The problem is in these parts of the code:

for y_pred in y_preds:
    min_dist = np.abs(y_trues - y_pred).min()
    precision_counter += (min_dist <= self.tolerance)

for y_true in y_trues:
    min_dist = np.abs(y_preds - y_true).min()
    recall_counter += (min_dist <= self.tolerance)

The issue here is that this method of counting can lead to overcounting in both precision and recall calculations:

  • For precision: Each prediction is compared to all ground truths, and if any ground truth is within the tolerance, it's counted. This means that multiple predictions could be counted as correct for the same ground truth value.
  • For recall: Each ground truth is compared to all predictions, and if any prediction is within the tolerance, it's counted. This means that multiple ground truths could be counted as correctly predicted by the same prediction.

Updated results for TIMIT:

saving for test - cpc_1 - {'prominence': 2.5e-05, 'width': None, 'distance': None, 'epoch': 20}
        train_nfc_loss                 -- 0
        test_nfc_loss                  -- 0.5646438481595352
        epoch                          -- 21
        current_lr                     -- 0
        timit_test_cpc_1_f1            -- 0.5590702254872026
        timit_test_cpc_1_precision     -- 0.5742684664073275
        timit_test_cpc_1_recall        -- 0.5446651824704964
        timit_test_cpc_1_rval          -- 0.6281234855558321
        timit_test_cpc_1_max_rval      -- 0.6281234855558321
        timit_test_cpc_1_max_rval_epoch -- 20
        test_max_rval                  -- 0.6281234855558321
------------------------------------------------------------------------------------------

Metrics overcounting fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant