Another kind of testing

[LemuelPuglisi]

First of all, I want to thank the author of the package for his great work.

I have a medical imaging dataset, where each image belongs to a patient. Each patient has at least 3/4 images inside the dataset. Hence, there isn't a fixed number of classes. The method I'm currently using to test my algorithms is described as follows:

1. Compute all the embeddings from the test set
2. Then for each (test) embedding: exclude the embedding from the set and run nearest neighbor

The question is: there is a way to replicate this behavior with the Tester class? Or do I need to extend the BaseTester and implement this method?

Thank you in advance.

[KevinMusgrave]

Since you're already computing all the embeddings, you can use AccuracyCalculator:

from pytorch_metric_learning.utils.accuracy_calculator import AccuracyCalculator

ac = AccuracyCalculator(exclude=("AMI", "NMI"))
accuracies = ac.get_accuracy(emb, emb, labels, labels, embeddings_come_from_same_source=True)
print(accuracies)

The knn computation uses faiss. If you don't want to install faiss, you can try this:

from pytorch_metric_learning.distances import LpDistance
from pytorch_metric_learning.utils.inference import CustomKNN

distance = LpDistance() #L2 normalized distance
knn_func = CustomKNN(distance)
ac = AccuracyCalculator(exclude=("AMI", "NMI"), knn_func=knn_func)

[LemuelPuglisi]

Thanks for your quick response. So embeddings_come_from_same_source flag takes care of excluding the query embedding from the kNN dataset when computing evaluation metrics?

[KevinMusgrave]

Yes

[LemuelPuglisi]

thanks!!

[KevinMusgrave]

The latest version (1.5.2) might interest you: https://github.com/KevinMusgrave/pytorch-metric-learning/releases/tag/v1.5.2