Baseline model in paper

[bkj]

Hi -- Have you ever tried to reproduce the baseline results reported in the paper? I implemented Matching Networks from scratch, and my baseline model does substantially better than the one reported in the paper (66% reduction in error). Am asking around to see whether other people have ever noticed a similar issue.

Thanks
Ben

[AntreasAntoniou]

What are the exact numbers you are getting vs the paper? In which dataset?

…

On Thu, 8 Nov 2018 at 22:42, Ben Johnson ***@***.***> wrote: Hi -- Have you ever tried to reproduce the baseline results reported in the paper? I implemented Matching Networks from scratch, and my baseline model does substantially better than the one reported in the paper (66% reduction in error). Am asking around to see whether other people have ever noticed a similar issue. Thanks Ben — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#16>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKSuNsM-2jxYu_LfsQxVgQJmAgUaz95Vks5utLNTgaJpZM4YVn-2> .

[bkj]

On Omniglot, 5way/1shot and 20way/1shot they report 86.0% and 72.9% accuracies, respectively. I'm getting something like 95% and 90% from the baseline on those tasks.

Note I'm not using the same split as them (I don't think it's available?). This is using the background dataset as training and the evaluation set as testing.

[AntreasAntoniou]

In the paper they report: MATCHING NETS (OURS) Cosine N 98.1% 98.9% 93.8% 98.5% for 5-way 5-shot, 5-way 1-shot, 20-way 1-shot and 20-way 5-shot respectively. Where did you get 86 and 72.9%?

…

On Thu, 8 Nov 2018 at 22:48, Ben Johnson ***@***.***> wrote: On Omniglot, 5way/1shot and 20way/1shot they report 86.0% and 72.9% accuracies, respectively. I'm getting something like 95% and 90% from the baseline on those tasks. Note I'm not using the same split as them (I don't think it's available?). This is using the background dataset as training and the evaluation set as testing. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKSuNiqP7JEYaNyqmYF1-Bz4hOEFyi42ks5utLTPgaJpZM4YVn-2> .

[bkj]

I'm talking about the baseline models, not the Matching Networks.

Model Matching Fn Fine Tune 5-way Acc 20-way Acc
1-shot 5-shot 1-shot 5-shot
PIXELS Cosine N 41.7% 63.2% 26.7% 42.6%
BASELINE CLASSIFIER Cosine N 80.0% 95.0% 69.5% 89.1%
BASELINE CLASSIFIER Cosine Y 82.3% 98.4% 70.6% 92.0%
BASELINE CLASSIFIER Softmax Y 86.0% 97.6% 72.9% 92.3% ***
MANN (NO CONV) [21] Cosine N 82.8% 94.9% – –
CONVOLUTIONAL SIAMESE NET [11] Cosine N 96.7% 98.4% 88.0% 96.5%
CONVOLUTIONAL SIAMESE NET [11] Cosine Y 97.3% 98.4% 88.1% 97.0%
MATCHING NETS (OURS) Cosine N 98.1% 98.9% 93.8% 98.5%
MATCHING NETS (OURS) Cosine Y 97.9% 98.7% 93.5% 98.7%

I'm talking about the ***