Reproducing benchmark results on Oxford 5k dataset

[insikk]

Thank you for sharing your source code. It is really helpful to promote CBIR research community.

From disclaimer you mentioned that this source code can be differ from the codes used for the paper publication. However, I observe huge difference.

The reported performance on Oxf105k and Par106k follows below:

	retrieval_rank	inliers	eff.inliers	inter-image	inter-place	inter-place+pop
oxf105k(mAP)	-	0.710	0.730	0.708	0.735	0.745
Par106k(mAP)	-	0.613	0.619	0.611	0.649	0.682

I tried to get results on Oxford 5k with 200k vocabulary trained on Paris 6k.

I got results which is weird for two reasons.

• I though the model should gives better result when we use oxford 5k without distractor 100k data, but it is worse than oxf105k
• Having better measure make the result worse. Do you have the same result? i.e. eff.inliers give worse result than just using retrieval_rank ?

	retrieval_rank	inliers	eff.inliers	inter-image	inter-place	inter-place+pop
oxf5k(mAP)	0.700	0.679	0.682	0.674	x	x

Maybe, I made some obvious mistake. I assume you had run a lot of experiment. If you have any clues, please give me a hint.

[tsattler]

@insikk Sorry for the late reply. This somehow slipped past my attention.

What type of features are you using? I recommend using an upright Hessian affine detector (https://github.com/perdoch/hesaff) with RootSIFT descriptors.

[wangmaoCS]

If I use the upright Hessian affine detector (https://github.com/perdoch/hesaff) with RootSIFT descriptors with the geometric_burstiness code, can I achieve the result reported in the CVPR paper?
I have tried my best but failed to achieve the reported mAP.

[tsattler]

I would assume that you get at the very least close to the results reported in the paper. In my experience, using the right type of features and a well-trained visual vocabulary makes a significant difference.

Please notice that the implementation that we released is not 100% the same as in our paper. In the paper, the retrieval and spatial verification were done using the pipeline that we released. The re-ranking was implemented in Matlab and I re-implemented this in C++ to make it easier to use. There might be some bugs in this part (I recently fixed one).
We also used a different feature detection and extraction method as well as a vocabulary provided by another researcher. We cannot share this data. But the upright Hessian detector with RootSIFT descriptors and a well-trained vocabulary should give you similar performance.