YCBV results have discrepancy with the BOP 2019 leaderboard.

[eliphatfs]

I ran run_ycb_video.py on the BOP 2019 test set. Compared to the BOP benchmark leaderboard, the AR VSD is a bit higher, but MSSD and MSPD are much worse.

Reproduced results using the official checkpoints:

{
    "bop19_average_recall": 0.6737796103161129,
    "bop19_average_recall_mspd": 0.5392190152801358,
    "bop19_average_recall_mssd": 0.6055299539170507,
    "bop19_average_recall_vsd": 0.876589861751152,
    "bop19_average_time_per_image": 1
}

BOP 2019 leaderboard results:

{
    "bop19_average_recall": 0.8890799579594146,
    "bop19_average_recall_mspd": 0.8803783652680087,
    "bop19_average_recall_mssd": 0.9305360174630124,
    "bop19_average_recall_vsd": 0.8563254911472229,
    "bop19_average_time_per_image": 9.94199268076155
}

I found that for objects that are symmetric in geometry but not symmetric in texture like cans in the first scene, the reproduced model sometimes gives a wrong rotation with large error (of 90 to 180 degrees) when the BOP 2019 submission result does not. Maybe related with #80.

I wonder if this is correct due to the missing diffusion augmentation in the released weights.

Also, I wonder what are the hyperparameters when submitting for the benchmark? I think on my 4090 it runs at about 1s/object but it is 9s on A100 in the BOP leaderboard so I think there is a discrepancy of hyperparameters.

Thank you in advance!

[OrestisVaggelis]

Hello, how did you run the evaluation after acquiring the .yml that is being produced?

[eliphatfs]

I wrote this script to convert the output yaml to bop19 csv:

import yaml
import numpy
import calibur


def arr2str(x: numpy.ndarray):
    return ' '.join(map(str, x.reshape(-1).tolist()))


x: dict[object, dict[object, dict]] = yaml.safe_load(open("ycbv_res.yml"))
with open("fp_ycbv-test.csv", "w") as fo:
    fo.write("scene_id,im_id,obj_id,score,R,t,time\n")
    for video_id in x.keys():
        for frame_id in x[video_id].keys():
            for obj_id in x[video_id][frame_id]:
                pose = numpy.array(x[video_id][frame_id][obj_id])
                # pose = numpy.linalg.inv(calibur.convert_pose(numpy.linalg.inv(pose), calibur.CC.GL, calibur.CC.CV))
                r = pose[:3, :3]
                t = pose[:3, 3] * 1000
                fo.write(f"{int(video_id)},{int(frame_id)},{int(obj_id)},1.0,{arr2str(r)},{arr2str(t)},1.0\n")

[LoveJu1y]

hello , would you please share some results of running time on bop challenge? why on the leaderboard of bop ,foundation pose needs 29 second per image ,while in my testing ,one instance just needs about 1s, are there that much instances in one images?

[aIacoella]

Hi!
I'm getting some similar discrepancies on mssd/mspd vs vsd.

Any news on why this could be?