DEBLEU example with TeacherMaskSoftmaxEmbeddingHelper and Triggers

docs/code/losses.rst

@@ -68,6 +68,14 @@ Entropy
 .. autofunction:: texar.losses.sequence_entropy_with_logits
 
 
+DEBLEU
+==================
+
+:hidden:`debleu`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. autofunction:: texar.losses.debleu
+
+
 Loss Utils
 ===========
 

docs/code/modules.rst

@@ -134,6 +134,11 @@ Decoders
 .. autoclass:: texar.modules.GumbelSoftmaxEmbeddingHelper
     :members:
 
+:hidden:`TeacherMaskSoftmaxEmbeddingHelper`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. autoclass:: texar.modules.TeacherMaskSoftmaxEmbeddingHelper
+    :members:
+
 :hidden:`get_helper`
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 .. autofunction:: texar.modules.get_helper

docs/code/utils.rst

@@ -278,3 +278,21 @@ AverageRecorder
 ==========================
 .. autoclass:: texar.utils.AverageRecorder
     :members:
+
+Trigger
+==========================
+
+:hidden:`Trigger`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. autoclass:: texar.utils.Trigger
+    :members:
+
+:hidden:`ScheduledStepsTrigger`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. autoclass:: texar.utils.ScheduledStepsTrigger
+    :members:
+
+:hidden:`BestEverConvergenceTrigger`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. autoclass:: texar.utils.BestEverConvergenceTrigger
+    :members:

examples/differentiable_expected_bleu/README.md

@@ -0,0 +1,48 @@
+# Seq2seq Model #
+
+This example builds an attentional seq2seq model for machine translation trained with Differentiable Expected BLEU (DEBLEU) and Teacher Mask. See https://openreview.net/pdf?id=S1x2aiRqFX for the implemented paper.
+
+### Dataset ###
+
+  * iwslt14: The benchmark [IWSLT2014](https://sites.google.com/site/iwsltevaluation2014/home) (de-en) machine translation dataset. 
+
+Download the data with the following cmds:
+
+```bash
+python prepare_data.py --data de-en
+```
+
+### Train the model ###
+
+Train the model with the following cmd:
+
+```bash
+python differentiable_expected_bleu.py --config_model config_model_medium --config_data config_data_iwslt14_de-en --config_train config_train --expr_name iwslt14_de-en --restore_from "" --reinitialize
+```
+
+Here:
+  * `--config_model` specifies the model config. Note not to include the `.py` suffix.
+  * `--config_data` specifies the data config.
+  * `--config_train` specifies the training config.
+  * `--expr_name` specifies the experiment name. Used as the directory name to save and restore all information.
+  * `--restore_from` specifies the checkpoint path to restore from. If not specified (or an empty string is specified), the latest checkpoint in `expr_name` is restored.
+  * `--reinitialize` is a flag indicates whether to reinitialize the state of the optimizers before training and after annealing. Default is enabled.
+
+[config_model_medium.py](./config_model_medium.py) specifies a single-layer seq2seq model with Luong attention and bi-directional RNN encoder.
+
+[config_model_large.py](./config_model_large.py) specifies a seq2seq model with Luong attention, 2-layer bi-directional RNN encoder, single-layer RNN decoder, and a connector between the final state of the encoder and the initial state of the decoder. The size of this model is quite large.
+
+[config_data_iwslt14_de-en.py](./config_data_iwslt14_de-en.py) specifies the IWSLT'14 German-English dataset.
+
+[config_train.py](./config_train.py) specifies the training (including annealing) configs.
+
+## Results ##
+
+On the IWSLT'14 German-English dataset, we ran both configs for 4~5 times. Here are the average BLEU scores attained:
+
+|                       config                       | inference beam size | Cross-Entropy baseline | DEBLEU | improvement |
+| :------------------------------------------------: | :-----------------: | :--------------------: | :----: | :---------: |
+| [config_model_medium.py](./config_model_medium.py) |          1          |         26.12          | 27.40  |    1.28     |
+| [config_model_medium.py](./config_model_medium.py) |          5          |         27.03          | 27.72  |    0.70     |
+|  [config_model_large.py](./config_model_large.py)  |          1          |         25.24          | 26.47  |    1.23     |
+|  [config_model_large.py](./config_model_large.py)  |          5          |         26.33          | 26.87  |    0.54     |

examples/differentiable_expected_bleu/config_data_iwslt14_de-en.py

@@ -0,0 +1,59 @@
+source_vocab_file = 'data/iwslt14_de-en/vocab.de'
+target_vocab_file = 'data/iwslt14_de-en/vocab.en'
+
+train_0 = {
+    'batch_size': 80,
+    'allow_smaller_final_batch': False,
+    'source_dataset': {
+        "files": 'data/iwslt14_de-en/train.de',
+        'vocab_file': source_vocab_file,
+        'max_seq_length': 50
+    },
+    'target_dataset': {
+        'files': 'data/iwslt14_de-en/train.en',
+        'vocab_file': target_vocab_file,
+        'max_seq_length': 50
+    },
+}
+
+train_1 = {
+    'batch_size': 160,
+    'allow_smaller_final_batch': False,
+    'source_dataset': {
+        "files": 'data/iwslt14_de-en/train.de',
+        'vocab_file': source_vocab_file,
+        'max_seq_length': 50
+    },
+    'target_dataset': {
+        'files': 'data/iwslt14_de-en/train.en',
+        'vocab_file': target_vocab_file,
+        'max_seq_length': 50
+    },
+}
+
+
+val = {
+    'batch_size': 80,
+    'shuffle': False,
+    'source_dataset': {
+        "files": 'data/iwslt14_de-en/valid.de',
+        'vocab_file': source_vocab_file,
+    },
+    'target_dataset': {
+        'files': 'data/iwslt14_de-en/valid.en',
+        'vocab_file': target_vocab_file,
+    },
+}
+
+test = {
+    'batch_size': 80,
+    'shuffle': False,
+    'source_dataset': {
+        "files": 'data/iwslt14_de-en/test.de',
+        'vocab_file': source_vocab_file,
+    },
+    'target_dataset': {
+        'files': 'data/iwslt14_de-en/test.en',
+        'vocab_file': target_vocab_file,
+    },
+}

examples/differentiable_expected_bleu/config_data_iwslt14_en-fr.py

@@ -0,0 +1,45 @@
+source_vocab_file = 'data/iwslt14_en-fr/vocab.en'
+target_vocab_file = 'data/iwslt14_en-fr/vocab.fr'
+
+batch_size = 80
+
+train = {
+    'batch_size': batch_size,
+    'allow_smaller_final_batch': False,
+    'source_dataset': {
+        "files": 'data/iwslt14_en-fr/train.en',
+        'vocab_file': source_vocab_file,
+        'max_seq_length': 50
+    },
+    'target_dataset': {
+        'files': 'data/iwslt14_en-fr/train.fr',
+        'vocab_file': target_vocab_file,
+        'max_seq_length': 50
+    },
+}
+
+val = {
+    'batch_size': batch_size,
+    'shuffle': False,
+    'source_dataset': {
+        "files": 'data/iwslt14_en-fr/valid.en',
+        'vocab_file': source_vocab_file,
+    },
+    'target_dataset': {
+        'files': 'data/iwslt14_en-fr/valid.fr',
+        'vocab_file': target_vocab_file,
+    },
+}
+
+test = {
+    'batch_size': batch_size,
+    'shuffle': False,
+    'source_dataset': {
+        "files": 'data/iwslt14_en-fr/test.en',
+        'vocab_file': source_vocab_file,
+    },
+    'target_dataset': {
+        'files': 'data/iwslt14_en-fr/test.fr',
+        'vocab_file': target_vocab_file,
+    },
+}

examples/differentiable_expected_bleu/config_model_large.py

@@ -0,0 +1,39 @@
+# Attentional Seq2seq model.
+# Hyperparameters not specified here will take the default values.
+
+num_units = 1000
+embedding_dim = 500
+
+embedder = {
+    'dim': embedding_dim
+}
+
+encoder = {
+    'rnn_cell_fw': {
+        'kwargs': {
+            'num_units': num_units
+        },
+        'num_layers': 2
+    },
+    'output_layer_fw': {
+        'dropout_rate': 0
+    }
+}
+
+connector = {
+    'activation_fn': 'tanh'
+}
+
+decoder = {
+    'rnn_cell': {
+        'kwargs': {
+            'num_units': num_units
+        },
+    },
+    'attention': {
+        'kwargs': {
+            'num_units': num_units,
+        },
+        'attention_layer_size': num_units
+    }
+}

examples/differentiable_expected_bleu/config_model_medium.py

@@ -0,0 +1,40 @@
+# Attentional Seq2seq model.
+# Hyperparameters not specified here will take the default values.
+
+num_units = 256
+embedding_dim = 256
+dropout = 0.2
+
+embedder = {
+    'dim': embedding_dim
+}
+
+encoder = {
+    'rnn_cell_fw': {
+        'kwargs': {
+            'num_units': num_units
+        },
+        'dropout': {
+            'input_keep_prob': 1. - dropout
+        }
+    }
+}
+
+connector = None
+
+decoder = {
+    'rnn_cell': {
+        'kwargs': {
+            'num_units': num_units
+        },
+        'dropout': {
+            'input_keep_prob': 1. - dropout
+        }
+    },
+    'attention': {
+        'kwargs': {
+            'num_units': num_units,
+        },
+        'attention_layer_size': num_units
+    }
+}

examples/differentiable_expected_bleu/config_train.py

@@ -0,0 +1,80 @@
+max_epochs = 1000
+steps_per_eval = 500
+tau = 1.
+infer_beam_width = 1
+infer_max_decoding_length = 50
+
+threshold_steps = 10000
+minimum_interval_steps = 10000
+phases = [
+    # (config_data, config_train, mask_pattern)
+    ("train_0", "xe_0", None),
+    ("train_0", "xe_1", None),
+    ("train_0", "debleu_0", (2, 2)),
+    ("train_1", "debleu_0", (4, 2)),
+    ("train_1", "debleu_1", (1, 0)),
+]
+
+train_xe_0 = {
+    "optimizer": {
+        "type": "AdamOptimizer",
+        "kwargs": {
+            "learning_rate": 1e-3
+        }
+    },
+    "gradient_clip": {
+        "type": "clip_by_global_norm",
+        "kwargs": {
+            "clip_norm": 5.
+        }
+    },
+    "name": "XE_0"
+}
+
+train_xe_1 = {
+    "optimizer": {
+        "type": "AdamOptimizer",
+        "kwargs": {
+            "learning_rate": 1e-5
+        }
+    },
+    "gradient_clip": {
+        "type": "clip_by_global_norm",
+        "kwargs": {
+            "clip_norm": 5.
+        }
+    },
+    "name": "XE_1"
+}
+
+train_debleu_0 = {
+    "optimizer": {
+        "type": "AdamOptimizer",
+        "kwargs": {
+            "learning_rate": 1e-5
+        }
+    },
+    "gradient_clip": {
+        "type": "clip_by_global_norm",
+        "kwargs": {
+            "clip_norm": 5.
+        }
+    },
+    "name": "DEBLEU_0"
+}
+
+train_debleu_1 = {
+    "optimizer": {
+        "type": "AdamOptimizer",
+        "kwargs": {
+            "learning_rate": 1e-6
+        }
+    },
+    "gradient_clip": {
+        "type": "clip_by_global_norm",
+        "kwargs": {
+            "clip_norm": 5.
+        }
+    },
+    "name": "DEBLEU_1"
+}