MeanAveragePrecision (map) returns -1.0 if max_detection_thresholds is changed

[marckatzenmaier]

🐛 Bug

I found a bug when changing the max_detection_thresholds of MeanAveragePrecision to a large number e.g. [1, 10, 1000] the result of the map calculation becomes -1.0
I wanted to increase the number of detections per image since my data has up to several thousand instances per image. I detected that the same problem occurs with the example given in the class definition.

To Reproduce

Run the example from the function definition with max_detection_thresholds=[1, 10, 1000] instead of the default [1, 10, 100].

Code sample

import torch
from torchmetrics.detection.mean_ap import MeanAveragePrecision
preds = [dict(
              boxes = torch.tensor([[258.0, 41.0, 606.0, 285.0]]),
              scores = torch.tensor([0.536]),
              labels = torch.tensor([0]),
             )]
target = [dict(
               boxes = torch.tensor([[214.0, 41.0, 562.0, 285.0]]),
               labels = torch.tensor([0]),
             )]
metric = MeanAveragePrecision(max_detection_thresholds=[1, 10, 1000])
metric.update(preds, target)
from pprint import pprint
pprint(metric.compute())

Output:

{'map': tensor(-1.),
 'map_50': tensor(1.),
 'map_75': tensor(1.),
 'map_large': tensor(0.6000),
 'map_medium': tensor(-1.),
 'map_per_class': tensor(-1.),
 'map_small': tensor(-1.),
 'mar_1': tensor(0.6000),
 'mar_10': tensor(0.6000),
 'mar_1000': tensor(0.6000),
 'mar_1000_per_class': tensor(-1.),
 'mar_large': tensor(0.6000),
 'mar_medium': tensor(-1.),
 'mar_small': tensor(-1.)}

Expected behavior

Get the same output as with [1, 10, 100]:

{'map': tensor(0.6000),
 'map_50': tensor(1.),
 'map_75': tensor(1.),
 'map_large': tensor(0.6000),
 'map_medium': tensor(-1.),
 'map_per_class': tensor(-1.),
 'map_small': tensor(-1.),
 'mar_1': tensor(0.6000),
 'mar_10': tensor(0.6000),
 'mar_100': tensor(0.6000),
 'mar_100_per_class': tensor(-1.),
 'mar_large': tensor(0.6000),
 'mar_medium': tensor(-1.),
 'mar_small': tensor(-1.)}

Environment

• python 3.9
• pytorch: 1.10
• TorchMetrics: 0.9.0

Thanks for your help.

[github-actions]

Hi! thanks for your contribution!, great first issue!

[SkafteNicki]

Closed by #1712 .

[RocketFlash]

Same issue using last library version

[RocketFlash]

@SkafteNicki could you recommend what to do ?

[SkafteNicki]

@SkafteNicki could you recommend what to do ?

@RocketFlash Let me reopen the issue and investigate. A lot has happened to the implementation since the issue was first opened, so hard to say right now.

[RocketFlash]

@SkafteNicki it seems something wrong with pycocotools, I've tried faster version of COCOEval wrapper based on
facebook's detectron2 implementation

This one and it calculates mAP correctly (not -1)

[Borda]

it seems something wrong with pycocotools

What version are using as the original repo was forked and some OSS development continued