demo/experimentation_testing_no_need_to_train_fusion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Experimentation: Testing models that do not require training a fusion model (no learning to rank)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Two things to do before we start:\n",
    "1. Point environment variable `COLLECT_ROOT` to the collection root.\n",
    "2. Change directory to the location of installed scripts/binaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: COLLECT_ROOT=/home/leo/flexneuart_collections\n"
     ]
    }
   ],
   "source": [
    "%env COLLECT_ROOT=/home/leo/flexneuart_collections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/leo/flexneuart_scripts\n"
     ]
    }
   ],
   "source": [
    "cd /home/leo/flexneuart_scripts/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing BM25\n",
    "We use optimal BM25 parameters obtained during tuning:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using collection root: /home/leo/flexneuart_collections\n",
      "The number of CPU cores:      8\n",
      "The number of || experiments: 1\n",
      "The number of threads:        8\n",
      "================================================================================\n",
      "Experiment descriptor file:                                 /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/exper_desc.best/bm25.json\n",
      "Default test set:                                           dev\n",
      "Number of parallel experiments:                             1\n",
      "Number of threads in feature extractors/query applications: 8\n",
      "================================================================================\n",
      "Parsed experiment parameters:\n",
      "experSubdir:final_exper/bm25\n",
      "candProvAddConf:exper_desc.best/lucene.json\n",
      "extrTypeFinal:exper_desc.best/extractors/bm25.json\n",
      "modelFinal:exper_desc.best/models/one_feat.model\n",
      "testOnly:1\n",
      "========================================\n",
      "Started a process 8306, working dir: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25\n",
      "Process log file: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25/exper.log\n",
      "Waiting for 1 child processes\n",
      "Process with pid=8306 finished successfully.\n",
      "Waiting for 0 child processes\n",
      "================================================================================\n",
      "1 experiments executed\n",
      "0 experiments failed\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "!./exper/run_experiments.sh \\\n",
    "  wikipedia_dpr_nq_sample \\\n",
    "  exper_desc.best/bm25.json \\\n",
    "  -test_part dev"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the results are available in the directory `collections/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25`. \n",
    "\n",
    "The following is a summary report (top-100):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of queries:    2500\r\n",
      "NDCG@10:  0.400400\r\n",
      "NDCG@20:  0.433900\r\n",
      "NDCG@100: 0.507400\r\n",
      "P20:      0.164800\r\n",
      "MAP:      0.346200\r\n",
      "MRR:      0.487300\r\n",
      "Recall:   0.817827\r\n"
     ]
    }
   ],
   "source": [
    "!cat $COLLECT_ROOT/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25/rep/out_100.rep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sanity check: testing if the intermediate re-ranker functionality works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using collection root: /home/leo/flexneuart_collections\n",
      "The number of CPU cores:      8\n",
      "The number of || experiments: 1\n",
      "The number of threads:        8\n",
      "================================================================================\n",
      "Experiment descriptor file:                                 /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/exper_desc.best/bm25_test_interm.json\n",
      "Default test set:                                           dev\n",
      "Number of parallel experiments:                             1\n",
      "Number of threads in feature extractors/query applications: 8\n",
      "================================================================================\n",
      "Parsed experiment parameters:\n",
      "experSubdir:final_exper/bm25_test_interm\n",
      "candProvAddConf:exper_desc.best/lucene.json\n",
      "extrTypeInterm:exper_desc.best/extractors/bm25.json\n",
      "modelInterm:exper_desc.best/models/one_feat.model\n",
      "candProvQty:5000\n",
      "extrTypeFinal:exper_desc.best/extractors/bm25.json\n",
      "modelFinal:exper_desc.best/models/one_feat.model\n",
      "testOnly:1\n",
      "========================================\n",
      "Started a process 8439, working dir: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25_test_interm\n",
      "Process log file: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25_test_interm/exper.log\n",
      "Waiting for 1 child processes\n",
      "Process with pid=8439 finished successfully.\n",
      "Waiting for 0 child processes\n",
      "================================================================================\n",
      "1 experiments executed\n",
      "0 experiments failed\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "# The results should be the same as for the BM25 re-ranker\n",
    "!./exper/run_experiments.sh \\\n",
    "  wikipedia_dpr_nq_sample \\\n",
    "  exper_desc.best/bm25_test_interm.json \\\n",
    "  -test_part dev \\\n",
    "  -test_cand_qty_list 100,200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Top-100 results should be the same as for BM25 re-ranker:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of queries:    2500\r\n",
      "NDCG@10:  0.400600\r\n",
      "NDCG@20:  0.434100\r\n",
      "NDCG@100: 0.507700\r\n",
      "P20:      0.164800\r\n",
      "MAP:      0.346400\r\n",
      "MRR:      0.487700\r\n",
      "Recall:   0.817879\r\n"
     ]
    }
   ],
   "source": [
    "!cat $COLLECT_ROOT/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25_test_interm/rep/out_100.rep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing dense retrieval (ANCE) in the re-ranking mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using collection root: /home/leo/flexneuart_collections\n",
      "The number of CPU cores:      8\n",
      "The number of || experiments: 1\n",
      "The number of threads:        8\n",
      "================================================================================\n",
      "Experiment descriptor file:                                 /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/exper_desc.best/ance.json\n",
      "Default test set:                                           dev\n",
      "Number of parallel experiments:                             1\n",
      "Number of threads in feature extractors/query applications: 8\n",
      "================================================================================\n",
      "Parsed experiment parameters:\n",
      "experSubdir:final_exper/ance\n",
      "candProvAddConf:exper_desc.best/lucene.json\n",
      "extrTypeFinal:exper_desc.best/extractors/ance.json\n",
      "modelFinal:exper_desc.best/models/one_feat.model\n",
      "testOnly:1\n",
      "========================================\n",
      "Started a process 8537, working dir: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/ance\n",
      "Process log file: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/ance/exper.log\n",
      "Waiting for 1 child processes\n",
      "Process with pid=8537 finished successfully.\n",
      "Waiting for 0 child processes\n",
      "================================================================================\n",
      "1 experiments executed\n",
      "0 experiments failed\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "!./exper/run_experiments.sh \\\n",
    "  wikipedia_dpr_nq_sample \\\n",
    "      exper_desc.best/ance.json \\\n",
    "  -test_part dev"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To-100 report:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of queries:    2500\r\n",
      "NDCG@10:  0.649100\r\n",
      "NDCG@20:  0.651700\r\n",
      "NDCG@100: 0.692300\r\n",
      "P20:      0.152200\r\n",
      "MAP:      0.555200\r\n",
      "MRR:      0.865000\r\n",
      "Recall:   0.639296\r\n"
     ]
    }
   ],
   "source": [
    "!cat $COLLECT_ROOT/wikipedia_dpr_nq_sample/results/dev/final_exper/ance/rep/out_100.rep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing dense retrieval (averaged glove embeddings) in the re-ranking mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using collection root: /home/leo/flexneuart_collections\n",
      "The number of CPU cores:      8\n",
      "The number of || experiments: 1\n",
      "The number of threads:        8\n",
      "================================================================================\n",
      "Experiment descriptor file:                                 /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/exper_desc.best/avgembed.json\n",
      "Default test set:                                           dev\n",
      "Number of parallel experiments:                             1\n",
      "Number of threads in feature extractors/query applications: 8\n",
      "================================================================================\n",
      "Parsed experiment parameters:\n",
      "experSubdir:final_exper/avgembed\n",
      "candProvAddConf:exper_desc.best/lucene.json\n",
      "extrTypeFinal:exper_desc.best/extractors/avgembed.json\n",
      "modelFinal:exper_desc.best/models/one_feat.model\n",
      "testOnly:1\n",
      "========================================\n",
      "Experimental directory already exists (removing contents): /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/avgembed\n",
      "Cleaning the experimental directory: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/avgembed\n",
      "Started a process 9028, working dir: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/avgembed\n",
      "Process log file: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/avgembed/exper.log\n",
      "Waiting for 1 child processes\n",
      "Process with pid=9028 finished successfully.\n",
      "Waiting for 0 child processes\n",
      "================================================================================\n",
      "1 experiments executed\n",
      "0 experiments failed\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "!./exper/run_experiments.sh \\\n",
    "  wikipedia_dpr_nq_sample \\\n",
    "  exper_desc.best/avgembed.json \\\n",
    "  -test_part dev"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To-100 report:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of queries:    2500\r\n",
      "NDCG@10:  0.144400\r\n",
      "NDCG@20:  0.157500\r\n",
      "NDCG@100: 0.216500\r\n",
      "P20:      0.067500\r\n",
      "MAP:      0.101900\r\n",
      "MRR:      0.225200\r\n",
      "Recall:   0.419581\r\n"
     ]
    }
   ],
   "source": [
    "!cat $COLLECT_ROOT/wikipedia_dpr_nq_sample/results/dev/final_exper/avgembed/rep/out_100.rep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing a BERT ranking model\n",
    "One needs to start a query server that binds to the port 8080 as shown below. This needs to be done in __a separate terminal__, because notebooks do not support background processes. Please, note we have to specify __the same maximum query and document lengths__ as during the training process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "COLLECT_ROOT=/home/leo/flexneuart_collections\n",
    "\n",
    "./featextr_server/nn_rank_server.py  \\\n",
    "   --init_model $COLLECT_ROOT/wikipedia_dpr_nq_sample/derived_data/ir_models/vanilla_bert/model.best \\\n",
    "   --port 8080\n",
    "```\n",
    "\n",
    "Note that we ask to re-rank only 50 candidates. The ranking of candidates below 50th position will not change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using collection root: /home/leo/flexneuart_collections\n",
      "The number of CPU cores:      8\n",
      "The number of || experiments: 1\n",
      "The number of threads:        2\n",
      "================================================================================\n",
      "Experiment descriptor file:                                 /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/exper_desc.best/cedr8080.json\n",
      "Default test set:                                           dev\n",
      "Number of parallel experiments:                             1\n",
      "Number of threads in feature extractors/query applications: 2\n",
      "================================================================================\n",
      "Parsed experiment parameters:\n",
      "experSubdir:final_exper/cedr8080\n",
      "extrTypeFinal:exper_desc.best/extractors/cedr8080.json\n",
      "modelFinal:exper_desc.best/models/one_feat.model\n",
      "testOnly:1\n",
      "========================================\n",
      "Started a process 9373, working dir: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/cedr8080\n",
      "Process log file: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/cedr8080/exper.log\n",
      "Waiting for 1 child processes\n",
      "Process with pid=9373 finished successfully.\n",
      "Waiting for 0 child processes\n",
      "================================================================================\n",
      "1 experiments executed\n",
      "0 experiments failed\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "!./exper/run_experiments.sh \\\n",
    "  wikipedia_dpr_nq_sample \\\n",
    "  exper_desc.best/cedr8080.json \\\n",
    "  -thread_qty 2 \\\n",
    "  -max_final_rerank_qty 50 \\\n",
    "  -test_part dev "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Top-100 report:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of queries:    2500\r\n",
      "NDCG@10:  0.570000\r\n",
      "NDCG@20:  0.583500\r\n",
      "NDCG@100: 0.625800\r\n",
      "P20:      0.190800\r\n",
      "MAP:      0.497300\r\n",
      "MRR:      0.678000\r\n",
      "Recall:   0.808185\r\n"
     ]
    }
   ],
   "source": [
    "!cat $COLLECT_ROOT/wikipedia_dpr_nq_sample/results/dev/final_exper/cedr8080/rep/out_100.rep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}