Model Performance

Question Retrieval (Encoder-Cosine Similarity)

Model	Dev MAP	Dev MRR	Dev P@1	Dev P@5	Test MAP	Test MRR	Test P@1	Test P@5
LSTM	0.579	0.719	0.598	0.469	0.580	0.708	0.559	0.439
CNN	0.546	0.674	0.556	0.435	0.539	0.675	0.538	0.408

Domain Transfer (AUC 0.05)

Model	Dev	Test
Tf-Idf Similarity	0.707	0.739
Direct Transfer BiLSTM	0.568	0.540
Direct Transfer CNN	0.595	0.559
Adversarial BiLSTM	0.691	0.672
Adversarial CNN	0.705	0.668
Adversarial GRU	0.709	0.675

Environment

python >3.5, pytorch, a data_local folder should be present with data files. (AskUbuntu data in root folder extracted; Android data in root/Android folder extracted; glove.840B.300d.pruned.txt in root/glove folder, this file can be generated by preprocessing/generate_glove_pruned.py)

Evaluation

All *_eval.py files are for evaluating our trained best models. Simply run them using python.

Training

Other python files are for training. Simpy run them using python. Code are quite self explanatory, so tweaking parameters is fairly straightforward.

Reference

Lei, Tao, et al. “Semi-Supervised Question Retrieval with Gated Convolutions.” [1512.05726] Semi-Supervised Question Retrieval with Gated Convolutions, 4 Apr. 2016, arxiv.org/abs/1512.05726.

Ganin, Yaroslav, and Victor Lempitsky. “Unsupervised Domain Adaptation by Backpropagation.” [1409.7495] Unsupervised Domain Adaptation by Backpropagation, 27 Feb. 2015, arxiv.org/abs/1409.7495.

Zhang, Yuan, et al. “Aspect-Augmented Adversarial Networks for Domain Adaptation.” [1701.00188] Aspect-Augmented Adversarial Networks for Domain Adaptation, 25 Sept. 2017, arxiv.org/abs/1701.00188.

https://github.com/taolei87/askubuntu

https://github.com/jiangfeng1124/Android