DP_tf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DeepPavlov - an Open-Source Conversational AI Framework \n",
    "---\n",
    "\n",
    "<img align=\"center\" src=\"https://deeppavlov.ai/docs/_static/ipavlov_logo.png\"/>\n",
    "\n",
    "\n",
    "The open-source conversational AI framework [DeepPavlov](https://deeppavlov.ai/) offers a free and easy-to-use solution to build dialogue systems. DeepPavlov comes with a bunch of predefined components powered by [TensorFlow](https://www.tensorflow.org) and [Keras](https://keras.io) for solving NLP-related problems. The framework allows you to fine-tune hyperparameters and test models. You can check out the models in our [demo](https://demo.deeppavlov.ai)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "We support `Linux` and `Windows` platforms, `Python 3.6` and `Python 3.7`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install deeppavlov"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## QuickStart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [DeepPavlov](https://deeppavlov.ai/) NLP pipelines are defined in the separate configuration files under the *config/faq* folder. List of models is available on\n",
    "[the doc page](http://docs.deeppavlov.ai/en/master/features/overview.html)\n",
    "\n",
    "When you are decided on the model and a configuration file, there are two ways to use it\n",
    "\n",
    "* via **Command Line Interface (CLI)**\n",
    "* via **Python**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Command Line Interface (CLI)\n",
    "\n",
    "To get predictions from a model interactively through CLI, run\n",
    "\n",
    "```bash \n",
    "    python -m deeppavlov interact <config_path> [-d]\n",
    "```\n",
    "\n",
    "* `-d` downloads required data -- pretrained model files and embeddings\n",
    "  (optional).\n",
    "\n",
    "You can train it in the same simple way:\n",
    "\n",
    "```bash\n",
    "    python -m deeppavlov train <config_path> [-d]\n",
    "```\n",
    "\n",
    "To train on your own data you need to modify dataset reader path in the\n",
    "[train config doc](http://docs.deeppavlov.ai/en/master/intro/config_description.html#train-config).\n",
    "The data format is specified in the corresponding model doc page. \n",
    "\n",
    "There are even more actions you can perform with configs:\n",
    "\n",
    "```bash \n",
    "    python -m deeppavlov <action> <config_path> [-d]\n",
    "```\n",
    "\n",
    "* `<action>` can be\n",
    "    * `download` to download model's data (same as `-d`),\n",
    "    * `train` to train the model on the data specified in the config file,\n",
    "    * `evaluate` to calculate metrics on the same dataset,\n",
    "    * `interact` to interact via CLI,\n",
    "    * `riseapi` to run a REST API server (see\n",
    "    [doc](http://docs.deeppavlov.ai/en/master/integrations/rest_api.html)),\n",
    "    * `interactbot` to run as a Telegram bot (see\n",
    "    [doc](http://docs.deeppavlov.ai/en/master/integrations/telegram.html)),\n",
    "    * `interactmsbot` to run a Miscrosoft Bot Framework server (see\n",
    "    [doc](http://docs.deeppavlov.ai/en/master/integrations/ms_bot.html)),\n",
    "    * `predict` to get prediction for samples from *stdin* or from\n",
    "      *<file_path>* if `-f <file_path>` is specified.\n",
    "* `<config_path>` specifies path (or name) of model's config file\n",
    "* `-d` downloads required data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python\n",
    "\n",
    "To get predictions from a model interactively through Python, run\n",
    "\n",
    "```python \n",
    "    from deeppavlov import build_model\n",
    "\n",
    "    model = build_model(<config_path>, download=True)\n",
    "\n",
    "    # get predictions for 'input_text1', 'input_text2'\n",
    "    model(['input_text1', 'input_text2'])\n",
    "```\n",
    "\n",
    "* where `download=True` downloads required data from web -- pretrained model\n",
    "  files and embeddings (optional),\n",
    "* `<config_path>` is path to the chosen model's config file (e.g.\n",
    "  `\"deeppavlov/configs/ner/ner_ontonotes_bert_mult.json\"`) or\n",
    "  `deeppavlov.configs` attribute (e.g.\n",
    "  `deeppavlov.configs.ner.ner_ontonotes_bert_mult` without quotation marks).\n",
    "\n",
    "You can train it in the same simple way:\n",
    "\n",
    "```python \n",
    "    from deeppavlov import train_model \n",
    "\n",
    "    model = train_model(<config_path>, download=True)\n",
    "```\n",
    "\n",
    "* `download=True` downloads pretrained model, therefore the pretrained\n",
    "model will be, first, loaded and then train (optional).\n",
    "\n",
    "Dataset will be downloaded regardless of whether there was ``-d`` flag or\n",
    "not.\n",
    "\n",
    "To train on your own data you need to modify dataset reader path in the\n",
    "[train config doc](http://docs.deeppavlov.ai/en/master/intro/config_description.html#train-config).\n",
    "The data format is specified in the corresponding model doc page. \n",
    "\n",
    "You can also calculate metrics on the dataset specified in your config file:\n",
    "\n",
    "```python\n",
    "    from deeppavlov import evaluate_model \n",
    "\n",
    "    model = evaluate_model(<config_path>, download=True)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The config file consists of four main sections: **dataset_reader**, **dataset_iterator**, **chainer**, and **train**. The **dataset_reader** defines the dataset’s location along with the dataset format. After loading, the data is split into the train, validation, and test sets according to the **dataset_iterator** settings. The **chainer** section of the configuration files consists of three subsections. The **in** and **out** sections define an input and an output to the chainer, whereas the **pipe** section defines a pipeline of the required components to interact with the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load https://raw.githubusercontent.com/deepmipt/DeepPavlov/master/deeppavlov/configs/faq/tfidf_logreg_en_faq.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BERT-based Models of DeepPavlov\n",
    "\n",
    "---\n",
    "\n",
    "The release of [BERT](https://arxiv.org/abs/1810.04805) (Bidirectional Encoder Representations from Transformers) made the year 2018 an inflection point for the Natural Language Processing community. BERT is a transformer-based technique for pretraining language representations, which produces state-of-the-art results across a wide array of NLP tasks. BERT has been uploaded to TensorFlow Hub and offers seamless integration with DeepPavlov. We integrated BERT into three downstream tasks: **text classification**, **named entity recognition** (and sequence tagging in general), and **question answering**. As a result, we achieved substantial improvements in all these tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BERT for Text Classification\n",
    "\n",
    "We use the DeepPavlov BERT-based text classification model to the sentiment analysis problem. It involves identifying a writer’s attitude toward a particular topic. The attitude could be *positive*, *negative*, and *neutral*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m deeppavlov install insults_kaggle_bert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m deeppavlov interact insults_kaggle_bert -d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov import configs, build_model\n",
    "model = build_model(configs.classifiers.insults_kaggle_bert, download=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model(['hey, how are you?', 'You are so dumb!'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to train model on your data you need to create configuration file and set up **data_path** to folder with *train.csv*, *valid.csv*, *test.csv* and change **MODEL_PATH** where to save trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from pprint import pprint\n",
    "model_config = json.load(open(configs.classifiers.insults_kaggle_bert))\n",
    "\n",
    "pprint(model_config['dataset_reader'])\n",
    "pprint(model_config['metadata']['variables'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, train the model\n",
    "\n",
    "```bash\n",
    "    python -m deeppavlov train config_name\n",
    "```\n",
    "or in Python\n",
    "```python\n",
    "    from deeppavlov import train_model\n",
    "    model = train_model(model_config)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BERT for Named Entity Recognition\n",
    "\n",
    "Given a sequence of tokens (words, and possibly punctuation marks), provide a tag from a predefined tag set for each token in the sequence. For example, we want to extract persons' and organizations' names from the text. Then for the input text:\n",
    "\n",
    "    Yan Goodfellow works for Google Brain\n",
    "\n",
    "a NER model needs to provide the following sequence of tags:\n",
    "\n",
    "    B-PER I-PER    O     O   B-ORG  I-ORG\n",
    "\n",
    "Where *B-* and *I-* prefixes stand for the beginning and inside of the entity, while *O* stands for out of tag or no tag. Markup with the prefix scheme is called *BIO markup*. This markup is introduced for distinguishing of consequent entities with similar types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m deeppavlov install ner_ontonotes_bert_mult"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m deeppavlov interact ner_ontonotes_bert_mult -d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov import configs, build_model\n",
    "ner_model = build_model(configs.ner.ner_ontonotes_bert_mult, download=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_model(['Yan Goodfellow works for Google Brain'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multilingual NER"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov import configs, build_model\n",
    "ner_model_ml = build_model(configs.ner.ner_ontonotes_bert_mult, download=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_model_ml([\n",
    "\"Meteorologist Lachlan Stone said the snowfall in Queensland was an unusual occurrence \\\n",
    "  in a state with a sub-tropical to tropical climate.\",\n",
    "\"Церемония награждения пройдет 27 октября в развлекательном комплексе Hollywood and \\\n",
    "  Highland Center в Лос-Анджелесе (штат Калифорния, США).\", \n",
    "\"Das Orchester der Philharmonie Poznań widmet sich jetzt bereits zum zweiten \\\n",
    "  Mal der Musik dieses aus Deutschland vertriebenen Komponisten. Waghalter \\\n",
    "  stammte aus einer jüdischen Warschauer Familie.\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BERT for Question Answering\n",
    "\n",
    "Context question answering is the task of finding an answer to a question over a given context (e.g, a paragraph from Wikipedia), where the answer to each question is a segment of the context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m deeppavlov install squad_bert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m deeppavlov interact squad_bert -d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov import build_model, configs\n",
    "model_qa = build_model(configs.squad.squad_bert, download=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multilingual QA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov import build_model, configs\n",
    "model_qa_ml = build_model(configs.squad.squad_bert_multilingual_freezed_emb, download=True)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context_en = \"In meteorology, precipitation is any product of the condensation of atmospheric \\\n",
    "  water vapor that falls under gravity. The main forms of precipitation include drizzle, rain, \\\n",
    "  sleet, snow, graupel, and hail. Precipitation forms as smaller droplets coalesce via collision \\\n",
    "  with other raindrops or ice crystals within a cloud. Short, intense periods of rain in scattered locations \\\n",
    "  are called showers.\"\n",
    "\n",
    "context_fr = \"Les précipitations désignent tous les météores qui tombent dans une atmosphère \\\n",
    "  et il peut s'agir de solides ou de liquides selon la composition et la température de cette dernière. \\\n",
    "  Ce terme météorologique est le plus souvent au pluriel et désigne sur la Terre les hydrométéores \\\n",
    "  (cristaux de glace ou gouttelettes d'eau) qui, ayant été soumis à des processus de condensation \\\n",
    "  et d'agrégation à l'intérieur des nuages, sont devenus trop lourds pour demeurer en suspension \\\n",
    "  dans l'atmosphère et tombent au sol ou s'évaporent en virga avant de l'atteindre. Par extension, \\\n",
    "  le terme peut également être utilisé pour des phénomènes similaires sur d'autres planètes ou lunes ayant une atmosphère.\"\n",
    "\n",
    "model_qa_ml([context_en, context_fr, context_fr], \n",
    "      [\"Where do water droplets collide with ice crystals to form precipitation?\", \n",
    "       \"Sous quelle forme peut être précipitation?\", \n",
    "       \"Where the term precipitation can be used?\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Useful Links\n",
    "---\n",
    "\n",
    "[DeepPavlov Repository](https://github.com/deepmipt/DeepPavlov)\n",
    "\n",
    "[DeepPavlov Demo Page](https://demo.deeppavlov.ai)\n",
    "\n",
    "[DeepPavlov Documentation](https://docs.deeppavlov.ai)\n",
    "\n",
    "[Our Forum](https://forum.deeppavlov.ai)\n",
    "\n",
    "[Our Medium](https://medium.com/deeppavlov)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}