Generating multiple drafts of a book with different draft formatting settings in one experiment

[mmartin9684-sil]

With several new settings for controlling the formatting of drafts (paragraph marker placement, quotation denormalization, style marker placement), it would be convenient to be able to request drafts of the same book with different settings for these options. This would primarily be for internal testing of these new features, but also for purposes of introducing new projects to the impacts of these settings.

Currently, in order to create multiple drafts for the same book with different format settings, its necessary to make multiple runs of the experiment command. The first run would be for the full experiment (preprocess/train/test/translate) with one drafting configuration, followed by changes to the translate_config.yml, and then a second (partial) run of the experiment script (preprocess/translate or test/translate). In between the first and second runs of the experiment script, manual updates are need to the infer > checkpoint > source folder names, so that the first draft isn't overwritten by the second draft.

A single run of the experiment script that creates multiple drafts of the same book with varying drafting configurations would be simpler, less error prone, and less time-consuming.

[isaac091]

The first simple step towards something like this would be to save a copy of the draft with no post-processing done to it whenever some post-processing steps have been configured for.

Beyond that, we can have a discussion about what additional flexibility would be useful. The ability to have multiple translation configurations while only translating the text once would be new to silnlp, and so would take more work to implement, though it will probably be straightforward once we decide on a design that is intuitive for you all as users.

@davidbaines had also asked about being able to apply different post-processing after the fact. This would be relatively simple to implement as well, probably in a separate script.

I can knock out those couple of things and then we can discuss further as needed.

[isaac091]

Now that I think about it, a draft with no post-processing is not actually needed to produce another draft with alternate post-processing because, at least in the realm of USFM markers, the markers can be stripped out and placed back in again. So I'll hold off on making the translate step save a base draft, but once we have the more detailed configuration there will be a way to have one of the drafts be a base draft.

[isaac091]

@mmartin9684-sil @davidbaines

I added the script silnlp.common.postprocess_draft that isolates the marker placement process (and in the future other features under the 'Improving Draft Quality' umbrella) from the rest of the translate pipeline. The corresponding section of the wiki is here.

Unfortunately, you will still have to run it on ClearML to be able to use eflomal (if you're on Windows), so it won't be as big of a time saver as it hopefully will be in the future, but the groundwork is there.

There are two ways to use this script. The first is by passing the name of an experiment that has a translate config, in which case it will apply the postprocessing steps (given by the postprocess_draft arguments, not the translate config) to the book that was translated. If multiple books were translated or multiple translation requests were defined, it will only look at the first book in the first request, but I can expand the functionality as necessary. The second way is to pass file paths to the --source and --draft arguments, but this can only be done locally.

[isaac091]

On the topic of multiple configurations per draft, here is my proposal for the configuration:

translate:
  - src_project: NASB
    books: MAT
    postprocess:
      - include_paragraph_markers: False # no postprocessing
      - include_paragraph_markers: True  # only paragraph markers
      - include_paragraph_markers: True  # all markers
        include_style_markers: True
        include_embeds: True
  - src_project: NIV11R
    books: MAT
    postprocess:
      - include_paragraph_markers: True

[isaac091]

Suggestions from discussion today:

• Move the 'postprocess' section of the translate config to be at the same level as 'translate' and apply each configuration to each translate request. I agree that that makes more sense, and it also is more coherent from a larger design perspective to make the distinction between the translate step and the postprocessing step.
• Save a copy of each translated file with no postprocessing steps applied. As David pointed out, this eliminates the need for the non-intuitive include_paragraph_markers: False configuration, as well as allows for more flexibility when applying postprocessing steps after the initial translation run for types of postprocessing that are not as easy to undo, e.g. quote denormalization.