Replication of paper results

[gtesei]

Hello,

I would like to replicate the results in the paper. I used a simple bash script to generate the results.

#!/bin/bash

models=$(find trainedmodels/DS_COLING_2018_QA_models -name *.pkl)

for mod in $models
do
    echo ">> replicating results of $mod on websp..."
    python -m questionanswering.evaluate_on_test $mod  configs/webqsp_eval_config.yaml
    echo ">> replicating results of $mod on qald..."
    python -m questionanswering.evaluate_on_test $mod  configs/qald_eval_config_2.yaml
done

I got these results, that don't look like the ones in the paper.

STAGGModel_2018-03-14_811799,STAGGModel,Simple,-1,webqsp,full,NoEntityList,0.0,0.00968054211035818,0.0
STAGGModel_2018-03-14_811799,STAGGModel,Simple,container description not available,-1,webqsp,0,NoEntityList,0.0,0.002702702702702703,0.0,
STAGGModel_2018-03-14_811799,STAGGModel,Simple,container description not available,-1,webqsp,1,NoEntityList,0.0,0.013953488372093023,0.0,
STAGGModel_2018-03-14_811799,STAGGModel,Simple,container description not available,-1,webqsp,2,NoEntityList,0.0,0.04081632653061224,0.0,
STAGGModel_2018-03-14_811799,STAGGModel,Simple,container description not available,-1,webqsp,3,NoEntityList,0.0,0.10344827586206896,0.0,
STAGGModel_2018-03-14_811799,STAGGModel,Simple,-1,qald,full,NoEntityList,0.0,0.0,0.0
GNNModel_2018-03-15_369757,GNNModel,Simple,-1,webqsp,full,NoEntityList,0.0,0.00968054211035818,0.0
GNNModel_2018-03-15_369757,GNNModel,Simple,container description not available,-1,webqsp,0,NoEntityList,0.0,0.002702702702702703,0.0,
GNNModel_2018-03-15_369757,GNNModel,Simple,container description not available,-1,webqsp,1,NoEntityList,0.0,0.013953488372093023,0.0,
GNNModel_2018-03-15_369757,GNNModel,Simple,container description not available,-1,webqsp,2,NoEntityList,0.0,0.04081632653061224,0.0,
GNNModel_2018-03-15_369757,GNNModel,Simple,container description not available,-1,webqsp,3,NoEntityList,0.0,0.10344827586206896,0.0,
GNNModel_2018-03-15_369757,GNNModel,Simple,-1,qald,full,NoEntityList,0.0,0.0,0.0
PooledEdgesModel_2018-03-13_519272,PooledEdgesModel,Simple,-1,webqsp,full,NoEntityList,0.0,0.00968054211035818,0.0
PooledEdgesModel_2018-03-13_519272,PooledEdgesModel,Simple,container description not available,-1,webqsp,0,NoEntityList,0.0,0.002702702702702703,0.0,
PooledEdgesModel_2018-03-13_519272,PooledEdgesModel,Simple,container description not available,-1,webqsp,1,NoEntityList,0.0,0.013953488372093023,0.0,
PooledEdgesModel_2018-03-13_519272,PooledEdgesModel,Simple,container description not available,-1,webqsp,2,NoEntityList,0.0,0.04081632653061224,0.0,
PooledEdgesModel_2018-03-13_519272,PooledEdgesModel,Simple,container description not available,-1,webqsp,3,NoEntityList,0.0,0.10344827586206896,0.0,
PooledEdgesModel_2018-03-13_519272,PooledEdgesModel,Simple,-1,qald,full,NoEntityList,0.0,0.0,0.0
OneEdgeModel_2018-03-14_194679,OneEdgeModel,Simple,-1,webqsp,full,NoEntityList,0.0,0.00968054211035818,0.0
OneEdgeModel_2018-03-14_194679,OneEdgeModel,Simple,container description not available,-1,webqsp,0,NoEntityList,0.0,0.002702702702702703,0.0,
OneEdgeModel_2018-03-14_194679,OneEdgeModel,Simple,container description not available,-1,webqsp,1,NoEntityList,0.0,0.013953488372093023,0.0,
OneEdgeModel_2018-03-14_194679,OneEdgeModel,Simple,container description not available,-1,webqsp,2,NoEntityList,0.0,0.04081632653061224,0.0,
OneEdgeModel_2018-03-14_194679,OneEdgeModel,Simple,container description not available,-1,webqsp,3,NoEntityList,0.0,0.10344827586206896,0.0,
OneEdgeModel_2018-03-14_194679,OneEdgeModel,Simple,-1,qald,full,NoEntityList,0.0,0.0,0.0
GNNModel_2018-03-13_113541,GNNModel,Gated,-1,webqsp,full,NoEntityList,0.0,0.00968054211035818,0.0
GNNModel_2018-03-13_113541,GNNModel,Gated,container description not available,-1,webqsp,0,NoEntityList,0.0,0.002702702702702703,0.0,
GNNModel_2018-03-13_113541,GNNModel,Gated,container description not available,-1,webqsp,1,NoEntityList,0.0,0.013953488372093023,0.0,
GNNModel_2018-03-13_113541,GNNModel,Gated,container description not available,-1,webqsp,2,NoEntityList,0.0,0.04081632653061224,0.0,
GNNModel_2018-03-13_113541,GNNModel,Gated,container description not available,-1,webqsp,3,NoEntityList,0.0,0.10344827586206896,0.0,
GNNModel_2018-03-13_113541,GNNModel,Gated,-1,qald,full,NoEntityList,0.0,0.0,0.0

Any thought? ...

Thank you for any help.

PS I also modified questionanswering.evaluate_on_test replacing container.description with container description not available to avoid an error from fackel

[freyaya123]

have you changed the backend of Wikidata? like, backend: "http://localhost:8890/sparql"

[blackkingback]

Hey.Can I ask a question?What gpu did you use on this test?I keep getiing running out of memory error.