Ideas about how to keep the translations updated across versions (e.g. LTS and development)

[jonasraoni]

Summary

Our translation workflow for localized strings in OJS, OMP, OPS, pkp-lib and many plugins involves an integration between Weblate and Github. For each piece of software, Weblate is configured to base its work on a particular branch of a git repository. The translation updates are available as commits on a fork.

Unfortunately Weblate does not currently support multiple branches, so we must currently choose whether it operates on main or stable-3-3_0 (current LTS).

This leads to a problem where the translations are forked. If we point Weblate at stable-3_3_0, main falls out of date and will need to be reconciled; if we point Weblate at main, then users are frustrated with out-of-date translations on stable-3_3_0.

Conversation imported from Slack.

@asmecher
Unfortunately it's hard to port translations between 3.3 (LTS) and 3.4 -- there have been a lot of changes and keys moved.

@marcbria
An LTS without updated strings is a major handicap. In the past, with the first 3.3 we got a lot of complaints because the translations were not up to date and when we updated them, we had no way to package for the old versions (cherry-pick it is not trivial).

I guess the idea is keep the translations always up to date, but that's something we can't ask the translators to do because they mostly do this work on a voluntary basis.

I've given it some thought and (as weblate doesn't support different versions) the only thing I can think of is to have a second weblate against the LTS version... Is there any better solution?

Our team will keep the version we're on up to date (even if it's using the custom translations plugin), but it would be a shame if the work doesn't go back to the community.

I'm really interested to see how we can find a fit. If there is a sprint on this topic, is there a possibility to attend virtually?

@asmecher
Two Weblates are not a good solution, though; it means we'll have two competing versions of translations, which will be impossible to merge/reconcile.

@marcbria
I don't follow you Alec. 😅

One weblate will commit in "main" while the other installation will be setup to commit in "3.3-lts".

If the LTS is only modified with bug/security fixes and new translations, when would it compete with "main"?

@asmecher
I did pitch a discussion about this during the Finland sprint, and will probably try again in Bogota.

One weblate will commit in "main" while the other installation will be setup to commit in "3.3-lts".

If I'm following what you're proposing, this would have 2 big downsides:

We'd have to maintain double the Weblate environment (not the end of the world but a pain when adding/removing/managing translation components)

It would require translators to translate each string twice, and for competing translations to end up in different branches, leading to degraded quality when moving from an LTS to a newer branch.

[marcbria]

Until we find a real solution... Why not advice/mailing translators 2-3 weeks before each new release?

About weblate limitations so we know how much would cost to implement the "multi-branch" feature?

[marcbria]

To describe the issue from translator's perspective: one usual problem with translations is they are frozen when the release is packaged... and when the translator discovers a string is missing (usually when they install the packaged release or people report issues in translations), is not easy to fix it (because weblate is pointing to main branch, instead of the packaged one). And this problem also happens with LTS versions too.

I was working on a drupal and I noticed that the translations are downloaded independently of the core or plugin code... which allows both processes (code and translations) to follow different times. This way, a partially translated plugin/core can benefit from a translation that has been done months after its release.

Make sense to apply a similar approach in OxS tools and implement an "update locale" button?

I suspect it won't be a big difference for translators as they will need to commit to github branch directly instead of Weblate (that can only point to one branch) but will make final user's life easier.

Any idea about the weblate part, at least for the LTS versions?

[jonasraoni]

I guess it will be needed to spin up another Weblate instance at /v3.3.x or a sub-domain, unless it has very configurable hierarchies 🤔

About separating the localization from the code, and allowing the user to update it separately... These problems came into my mind:

• Translations that are not needed anymore cannot be removed. Otherwise it will break an old version of the software.
• Given the issue above, then we would need to switch from using locale keys to real text (e.g. __("Your password is invalid")), otherwise we'll end-up with translations that don't make sense (e.g. "Press the button X", when the button X doesn't exist yet).

[asmecher]

Starting up another parallel Weblate installation is a hard "no"; it guarantees that we'll double the amount of translation work required to keep the application up to date, and that adjustments between branches will be painful (due to missing or differently translated keys).

[marcbria]

Yes Alec. Your arguments against weblate duplication were clear in the initial post and are convincing.
The "summit" proposal you are making sounds really promising. Let's focus on this.
Thanks!

[asmecher]

Our current workflow

We may actually have 3 branches to consider. Imagine that 3.4 is released in Q1 2023, and 3.5 comes out in Q4 2023. As we approach the beginning of Q4, we will have:

• Current LTS (3.3) -- a large portion of the community will be using this
• Latest stable (3.4) -- a smaller portion, but active, will be using this
• Forthcoming stable (3.5) -- we'll want translation work to begin so that 3.5.0 can be released with solid translation support.

We can cherry-pick translation changes between branches. Our normal git workflow for merging the changes onto main is:

1. Add the remote (if you don't have it already). For OJS it would be:

   git remote add translations http://translate.pkp.sfu.ca/git/ojs/admin/
   

2. Fetch and merge the latest translations:

   git fetch translations
   git merge --no-ff translations/main
   

   (Very rarely it's necessary to resolve a merge conflict -- but I do ask that devs give me a heads-up before committing any major merges, in order that they can rebase and resolve within their work rather than the Weblate work.)

Because there is a merge commit, this results in the translations being grouped together in a way that's easier to merge to another branch. See this example commit. I have not tried merging to other branches, but it is a possibility.

Principles

Here are some principles I think are important to keep central.

Must-haves:

• Our general policy (with features and fixes) is that forward-porting from stable (especially LTS) towards unstable (especially main) is "safer" -- it is less likely to disrupt more conservative users. This could also apply to translations.
• Translators should not be expected to re-do work multiple times.
• It's important for both LTS and non-LTS releases to have good translations -- if languages are missing or out-of-date, it will discourage the use of the release.

Nice-to-haves:

• The translation files should become cleaner. There's currently a lot of unused content that wastes translator time.
• The technical management of translations should be as non-technical as possible. It would be great if we could reduce the use of manual git commands.

Proposal

I propose we use a "summit" workflow, as documented beautifully by the KDE project. This means:

• Maintaining a summit branch on each repository that contains an amalgamation of locale files from all supported branches. (For example, for the manager's interface in pkp-lib, there would be a summit branch on the pkp-lib repository that would contain a single manager.po file that includes content from the same file in both stable-3_3_0 and main branches, merged together.)
• Configuring Weblate to work directly on the summit branch. Translators would see content for all actively supported branches.
• Periodically "scattering" locale updates back to e.g. main and stable-3_3_0 branches, only updating content that is relevant to each branch.

The posummit tool appears to be designed just for the purpose. I suspect we can integrate it with all the necessary tools:

• Upon developer commit, either using git commit hooks (executed by the developer on their own machine) or github integration (executed upon pushing a commit to github). This would update the summit branch with any content that was changed/added by a developer.
• (Upon Weblate activity, so that translation updates can be automatically merged and scattered. I'm not sure this is necessary, or desirable while we don't have a formal translation review process in place.)

[marcbria]

I'll leave some questions/collateral ideas here (if you think it's not the right place, let me know):

• Has anyone explored free desktop translation tools like omegaT or poedit?

I mention this because maybe one of these tools can hide the git calls to the contributor-translator and offer a much simpler and more user-friendly ui/ux instead. And potentially, (accompanied by the respective documentation) this would allow translators to make contributions to specific versions.

And another question that comes to me from here is:

• As long as already released versions are frozen (no new tarballs are generated), doesn't that give more weight to the need for translations to be automatically updated as part of the install/upgrade process?

Eager to read your comments.

[asmecher]

@jonasraoni on possible conflicts when locale keys change meaning:

I guess we'll probably have problems with a locale key instructions translated as "Press the button X" on stable-3_3_0 and "Open the menu Y" on main. Does the tooling have support for such use case/conflict?

If a translation changes meaning between stable-3_3_0 and main without changing its locale key, then the posummit will recognize that the key exists in both places -- but choose one branch's translation to prioritize. Scattering the summited translations back to branches will cause the new translation to replace the old one in all branches the key appears in. Thus it's risky to have translations using the same locale key but with different meanings, and I suggest we avoid this by changing the locale key when the meaning changes. This would have the added benefit of allowing users to download and apply the summited locale directly to any supported version of the software.

Doing a test run of this on main vs. stable-3_3_0 actually doesn't show many conflicts -- just with email templates and URLs that changed from http to https. For email templates, we should change the key names; for the URL changes, back-porting these to main would be a feature, not a bug. And back-ports scattered to stable branches should get a spot review before a release is built.

I don't think Pology has tools for "monolingual PO", as it's called in the mode we're using it. (At least by Weblate.)

Another option would be to discuss if it's worth switching from locale keys to the original text.

This would make our use of PO files a little more standard, but we're not alone in using them the way that we are. Personally I think it's philosophically important for us to avoid Anglocentricism. It's nice to have all languages treated equally.

Moving back to the original text also wouldn't solve one of our biggest sources of mismatches between stable-3_3_0 and main: email templates. Presumably we wouldn't want to embed English email texts directly in the code, so we'd need another solution -- like the old XML files we used to support.

@marcbria on downloading updated translations:

I was working on a drupal and I noticed that the translations are downloaded independently of the core or plugin code... which allows both processes (code and translations) to follow different times. This way, a partially translated plugin/core can benefit from a translation that has been done months after its release.

As I alluded to above, this may be possible with the summited files if we take care to manage how we change locale key contents. I think we could start by solving the problem at hand in this discussion, and letting users manually download updated locales if they wish. If it turns out to be necessary to add download tools within the software, we can look at it. But I suspect this feature in Drupal is driven by the desire not to ship all languages with the download package, rather than making it easy to install updated translations.

@marcbria on workflows in other comparable tools:

...did you also research how other PHP-based projects (drupal, wordpress, nextcloud...) deal with this?

I looked both at what online translation tools support/recommend (Transifex and Weblate), and what Wordpress uses. Wordpress alludes to some kind of naive approach in its FAQ but I couldn't find better details. A translation tutorial I found appears to show different branches available for separate translation in what I suppose is a custom translation tool:

@marcbria on other points:

The technical management of translations should be as non-technical as possible. It would be great if we could reduce the use of manual git commands.

We can not ask our translator to learn git, so I will move this from "nice to have" to "Must have"... because any solution that don't take this limitation as a requirement won't be applied. It's not about "reduce the use of manual git commands", it's more a "no git commands at all".

Translators absolutely should not have to use git. I was referring to the work of the translation manager within PKP -- that's technically Emma, but I'm currently doing the work of merging translations from Weblate because the process is really laborious to do without working with git directly.

What do you mean by "formal translation review process"? It's a translation workflow based on privileges? (ie: community users: suggest, trusted translator: edit, notifications to trusted when new strings)

Currently anyone can register an account and make translation changes. Someone could presumably edit locale keys to include spam. Until we have a formalized review process that requires a second party to look over the changes before they're committed, translation changes shouldn't be automatically merged in.

I'm wondering if would be a good idea to ask plugin developers to avoid using shared keys.

It's always been possible for core translations to change meaning in ways that plugin authors can't predict, so this isn't a new problem introduced by this proposal. In practice, though, it hasn't happened because plugins tend to use really basic keys that don't change (submission.title) or they include their own locale keys. I propose we continue as now, with the added practice of changing locale keys when meanings change.

Could somebody remember me why we don't enable "Default translations" plugin by default?

That's outside the scope of this proposal :) but personally I think the default translation plugin hides the problem rather than solving it. I would rather have incomplete translations be visible.

Has anyone explored free desktop translation tools...

This is also out of scope :) but I've looked at a few of these and have no specific beef with them. However, going to a desktop translation model would 1) introduce a lot of additional steps/skills that translators would need to learn, and 2) make it a lot harder to contribute back or share translation work with other translators. Weblate takes care of all of this.

[marcbria]

You are absolutely right with my "out of scope". My apologies. ;-)

Back to the focus, from what I've seen both drupal and wordpress use custom made tools. Only wordpress has tried to do something a bit more generic with GlotPress but I think it doesn't provide anything that we don't already have the same or better with weblate.

For my part, one last thing before going ahead with your proposal Alec: someone took a look or have spoken to weblate to:

• Make sure that there is no solution to this?
• Know what it would cost to extend weblate to allow multiple branches?

I admit I haven't gone into it in depth but these two links look interesting:

• How do I translate several branches at once?
• Translation propagation

[NateWr]

I like the idea of controlling when we distribute translations into the main branch. One thing I've noticed is that with big new features, we have a lot of new locale strings. These then often get translated quickly after code is merged to the main branch. However, the text often undergoes substantial revision once it is in the main branch. This then creates extra work when changing these because we have to mark other translations as #, fuzzy. As we consolidate locale keys we move them between apps and that needs to be done for all locales. I'd love to get translations in the main branch only closer to release.

[asmecher]

WARNING WARNING WARNING: This documentation is outdated. See instead: https://github.com/asmecher/summit-tools/blob/main/README.md

Some experimentation with posummit:

Set up the environment

Directories etc.

mkdir summit # This is the "base" directory for later operations referred to as path/to/summit-base
cd summit-base
mkdir ojs
cd ojs
mkdir summit
mkdir branches
cd branches
git clone -b stable-3_3_0 [email protected]:pkp/ojs stable-3_3_0
git clone -b main [email protected]:pkp/ojs main

posummit configuration file

cd path/to/summit-base
cd ojs/summit

Add the configuration file summit-config-shared with contents:

S.over_templates = True

S.summit = dict(
    topdir=S.relpath("%s" % S.lang),
    topdir_templates=S.relpath("templates")
)

S.branches = [
    dict(id="main",
        topdir=S.relpath("../branches/main/locale/%s" % S.lang),
        topdir_templates=S.relpath("../branches/main/locale/templates")
    ),
    dict(id="stable-3_3_0",
        topdir=S.relpath("../branches/stable-3_3_0/locale/%s" % S.lang),
        topdir_templates=S.relpath("../branches/stable-3_3_0/locale/templates")
    ),
]

S.mappings = [
]

Disable msgfmt checking in posummit

Unfortunately posummit is hard-coded to run a set of checks on locale files that will cause a fatal error e.g. when msgid and translation don't both start with \n. (This is common for us because keys are short and synthetic, and translations are potentially long. It arises from our use of monolingual PO files, and is equally a problem encountered by other users of that Weblate configuration)

Edit posummit and find the line:

assert_system("msgfmt -c -o/dev/null %s " % tmp_path)

Comment it out with a #.

Create the .pot template files from the English translation

cd path/to/summit-base
cd ojs/branches/main/locale/en_US
mkdir ../templates
for file in `ls *.po | cut -d "." -f 1`; do msgfilter -i $file.po -o ../templates/$file.pot true; done
cd ../../../stable-3_3_0/locale/en_US
mkdir ../templates
for file in `ls *.po | cut -d "." -f 1`; do msgfilter -i $file.po -o ../templates/$file.pot true; done

Prepare a translation for the summiting process

This applies the .pot templates to the existing translation files. This is necessary in order to ensure that the translation has blank spots for all the necessary locale keys that might be scattered from the summit in later steps; if empty translations don't exist, posummit won't add them. It also has the benefit of turning outdated translations into comments, removing them from the translation pool in Weblate.

For cs_CZ in this example:

cd path/to/summit-base
cd ojs/branches/main/locale/cs_CZ
for file in `ls *.po | cut -d "." -f 1`; do msgmerge -N --previous --update $file.po ../templates/$file.pot; done

The resulting changes to the locale's .po files should be committed to the repository.

Creating the summit, gathering, merging, and scattering

cd path/to/summit-base
cd ojs/summit

# Collect the templates from the branches into a summited template
posummit summit-config-shared templates gather --create

# Collect the cs_CZ language from the branches into a summited translation
posummit summit-config-shared cs_CZ gather --create --force

# Merge and scatter the summited translations back to the branches
posummit summit-config-shared cs_CZ merge
posummit summit-config-shared cs_CZ scatter

Committing the results back to the repos

The scattering action above should result in changes to the checked-out branches. These can be reviewed, added, and committed.

Draft PRs

https://github.com/pkp/ojs/pull/3675/files

Features

Obsolete translations get commented out

This translation was at some point removed from the English locale file, but left hanging around cs_CZ. In the PR, it gets turned into a comment:

Translations with matching keys get scattered from main into stable-3_3_0

This translation is used in both main and stable-3_3_0, but the stable-3_3_0 translation was missing for cs_CZ. The msgmerge process made a blank spot for it in the appropriate stable-3_3_0 cs_CZ file, then the translation was scattered into it from main with posummit.

TODOs

• Need to update the posummit configuration to capture files outside of the locale/aa_BB directory (e.g. plugins). By default posummit wants to disambiguate files by filename; this may cause problems with multiple locale.po files (e.g. plugins).
• Need to deal with translations in e.g. stable-3_3_0 that should not be clobbered by main -- email templates where variables differ, system requirements, etc.
• Need to determine what to do with .pot files. Are these going to be throw-aways generated every time? Or committed somewhere and managed? (They're useful for traditional English-in-source-code use of gettext tools, but kind of redundant when using synthetic locale keys.)
• Need to investigate where/how #fuzzy is applied and whether it's appropriate.
• Need to consider policies on where/when translations are added. According to posummit documentation, translations should always be added at the summit, but that's assuming that the software is hard-coded in e.g. English and then translated afterwards.
• Need to figure out how this will scale with many repos (e.g. for plugins).
• Need to get a review on the test results in one or more languages.

WARNING WARNING WARNING: This documentation is outdated. See instead: https://github.com/asmecher/summit-tools/blob/main/README.md

[marcbria]

In this thread we were mainly focused in how WE (pkp and translators) deal with the translations but we talk a little about what final users expect.

Related with this, let me compile what is been said:

Marc: I was working on a drupal and I noticed that the translations are downloaded independently of the core or plugin code... which allows both processes (code and translations) to follow different times. This way, a partially translated plugin/core can benefit from a translation that has been done months after its release.
Make sense to apply a similar approach in OxS tools and implement an "update locale" button?

@jonasraoni About separating the localization from the code, and allowing the user to update it separately... These problems came into my mind:

• Translations that are not needed anymore cannot be removed. Otherwise it will break an old version of the software.
• Given the issue above, then we would need to switch from using locale keys to real text (e.g. __("Your password is invalid")), otherwise we'll end-up with translations that don't make sense (e.g. "Press the button X", when the button X doesn't exist yet).

Not sure if I was suggesting separate localizations from code (probably it's implicit but I was just thinking in the feature, not in the implementation) but what I discovered in the last meeting with our local network is most of them have broken translations and use default translations or custom translations plugins to deal with it.

Let me verbalize what is happening with the translation workflow so you can correct me if I'm thinking it in the wrong way...

Usually when new releases is packaged to be released, translations are not 100% complete (by design, only English does).
This is a fact and will always happen.

And I think it happens for many reasons: first because translators are volunteers and we can make them to do their work in the precise moment/windows we like them to translate, but also because translators work without context (don't know where a string will be placed in the interface) and this makes them fail or because they wrongly prioritize their work (we ask them to translate first that are more visible than the ones that are kind of hidden) and also because they are humans and... well we commit errors all the time.

So, at a certain time (months, but it could also be years... depending on the translation team upgrade plans) we got time to check the packaged new release, and we discover the missing chains so we go to weblate to translate them. But at this moment it's too late because a lot of fellows (newcomers or fellows that upgraded faster than us) downloaded a translation-broken version of OJS and won't probably upgrade for a while. And even if they upgrade fast, they will need to wait till next release to complete the missing chains.

So answering Jonas I thinks I understand the issues you pointed but:

It's not a big deal if they keep old translations till they upgrade, isn't it?
I mean, if we reset translations on every upgrade (except for the custom translation's plugin that will be keept and shown over the ones included in the release) old translations will be removed and the problem disappears.

I insist with the idea of the "upgrade button" because every time I talk with our local community (and also from the feedback got during the usability tests) the problem with the broken translations is perceived as a major issue.

@asmecher in my head this is all related and we need to think about it as a whole but if you prefer to keep the thread focused on finding a solution to keep different branches of translation or you think I'm straying too far off-topic, let me know and I'll move the debate where you say.

[nils-stefan-weiher]

From following the last posts does it make sense to track the "locale" Folder in the applcations with a Git submodule?
This would make updating the translations easier and it could also allow the creation of a little script to update the locales without upgrading the code.

[nils-stefan-weiher]

Or treat translation the same as plugins and use a "translation gallery" to manage them.

[marcbria]

Not sure if this discussion is still alive or we need to open a new conversation to talk about this part.

@nils-stefan-weiher I think the idea of the "translation gallery" is what @jonasraoni verbalized as "fresh localization updates without being forced to upgrade the installation" or I named as "upgrade button".

I really think that from the perspective of end users and for translators separate the translation of the release will be a big improvement.