feat: i18n framework based on extracting translations from plugin's /languages/ to /language/

[linonetwo]

Sorry for being late.

This is based on logic in #7821 (comment)

Lingo macro keep untouched. Will handle fallback between de-AT and de-DE. fixes #8263

How it works:

1. When wiki start and language change, $tw.wiki.unpackPluginTiddlers will be called, and in this method we can modify shadowTiddlers, which will be used by $tw.wiki.getTiddler from wikitext.
2. I add a call to my new $tw.utils.activatePluginTranslations method in unpackPluginTiddlers
3. Plugin authors will write l10n tiddlers under xxx/languages/, and this PR make TW auto move it to xxx/language/, so lingo macro works with xxx/language/ base path.
4. It move en-GB first, then the active language and dependeicies, so it will fallback in cascade.

1. RFC: $tw.utils.activatePluginTranslations will be undefined on first call, when utils is not mounted, so I check if it is undefined
2. RFC: when loading wiki and switch language, unpackPluginTiddlers will be called at least 3 times, wasting lots of repeated work in activatePluginTranslations. But I can't skip it, because it reset shadowTiddlers = Object.create(null); each time it's called.

Downside of this implementation is it not able to translate user wiki any more. Which was achieved by modify lingo macro in #7821:

!! Any l10n strings

To translate any text that directly placed in user's wiki, instead of in a language plugin, you can set the lingo-base variable to teach <<.def lingo>> macro the place to look for.

[github-actions]

Confirmed: linonetwo has already signed the Contributor License Agreement (see contributing.md)

[pmario]

I did just review the code. I'll give some more feedback. Once I did test it in detail.

[linonetwo]

I also tried patch one plugin from another, with fallback.

[pmario]

@linonetwo -- I thought you would have a closer look at the $:/core/modules/pluginswitcher.js, which is responsible to switch languages and themes.

I intended to suggest that the new functionality of extracting and removing languages should be part of the pluginswitcher.js

It is registered at startup.js but the main code is at /pluginswitcher.js

• At startup.js line 80 the language-switcher is registered, which in turn adds an event listener from pluginswitcher.js
• The swtichPlugins() funciton contains the logic to extract the language tiddlers from the corresponding plugins and overwrites the default en-GB shadow tiddlers, because they have the same "namespace"
• accumulatePlugin() takes care of the dependencies
• Then there is .unregisterPluginTiddlers()
• . registerPluginTiddlers()
• and finally .upackPluginTiddlers()

This mechanism makes sure that only those shadow tiddlers are in the shadow-store that are part of the activated language.
If a language is missing some translations, the default language or dependents are still there. Because of the same namespace.

IMO there is no reason to have all languages extracted from the plugins at startup. The number of language related shadow tiddlers should be constant and as low as possible.

We only have 1 core plugin in this PR, but the number of shadow tiddlers is increased by 174 tiddlers. The menubar-plugin de-DE translations.multids only contains 25 new texts + readme it should be 26 shadows more and not 174.

We have 33 possible languages in TW, which makes a total of 858 potential shadow tiddlers for menu-bar plugin alone.

The current total number of TW shadow tiddlers is: 2476 for v5.3.5 -- With the implementation in this PR the menubar has the potential to increase the number of shadows by 35%, which will directly impact the core performance for every filter that uses [all[shadows+tiddlers]] -- That's basically every UI related tiddler.

I was not happy, that every possible language is part of the plugin (because of size). -- But OK -- It was decided, that that's not a problem.

But I definitely think that only 1 set of language shadow tiddlers should be active at any given time.

[linonetwo]

every possible language is part of the plugin

In current implementation, you can use another language plugin to provide l10n. See my last comment.

• The swtichPlugins() funciton contains the logic to extract the language tiddlers

AFAIK It does this by calling unpackPluginTiddlers, which is I'm currently using. And in swtichPlugins() there is no way to access to shadowTiddlers. So I can't put my logic in it.

number of shadows will directly impact the core performance

In unpackPluginTiddlers -> activatePluginTranslations, I can delete unused shadowtiddlers.

[pmario]

number of shadows will directly impact the core performance

In unpackPluginTiddlers -> activatePluginTranslations, I can delete unused shadowtiddlers.

I think it should not be needed to delete unused shadow tiddlers. Only used shadows should be extracted into the /language/ namespace. As long as no unnecessary tiddlers are extracted no shadows need to be deleted.

en-GB will need to be always extracted since it is the "source of truth" and the final fallback. If the $:/language setting is de-DE then the German tiddlers are extracted into the same namespace that en-GB already uses. So no extra shadows are ever created. The same should be true for every other language, that is part of a plugin.

[linonetwo]

@pmario I've update the logic in unpackPluginTiddlers to not unpack unused language tiddlers.

The logic have to be called in unpackPluginTiddlers instead of in PluginSwitcher, because other places that call the unpackPluginTiddlers will clean up the language shadow tiddlers, so language shadow tiddlers added on language plugin switching/startup will lost.

I'm using $tw.language.activatePluginTranslations because I don't want to make boot.js file become much longer. Or we can use a library module and require it here, instead of use a util and check its existance.

[pmario]

@linonetwo

I do like your translation concept better than my initial one, which is much more complex. 🥇

I think the plugin menubar/language/ directory should be renamed to /languages/ to be consistent with the tiddler namespace.

The code and functionality looks good to me now.

every possible language is part of the plugin

In current implementation, you can use another language plugin to provide l10n. See my last comment.

I did miss that. That's a good thing. So we could decide to change the current behaviour if we want to, without compatibility problems.

@Jermolene

I did test:

• The dependents setting -> OK
  • The good thing is, that dependents only need to be defined once in the main language file. Plugins will inherit this configuration.
• Switching languages to German de-DE <-> de-AT -> OK
• de-AT only contains 2 translations -> uses fallback de-DE -> OK
• Change language -> Restart the wiki -> OK
• Change back to English -> OK
• Set language to de-DE and remove Config/HeadingMainXXX so en-GB fallback needs to be active -> OK

[pmario]

@linonetwo -- Please keep my de-AT and de-DE translations within this PR. So it's easier to test. -- I'll close my menubar-translation PR.

[linonetwo]

I think the plugin menubar/language/ directory should be renamed to /languages/ to be consistent with the tiddler namespace.

I was using /language/ because I wasn't delete unused shadow tiddlers. Now that they are deleted, it is OK to put then all under /languages/.

But notice that core translations are like {{$:/language/Buttons/Underline/Hint}}, and they are loaded from $:/languages/en-GB with /languages/. So it is also reasonable to keep the both /languages/ and /language/.

[pmario]

I was talking about the physical path name on the harddisk. The JS code is OK

[linonetwo]

@Jermolene @pmario Is current implementation good enough?

[pmario]

See my last post: #8435 (comment)

[linonetwo]

@pmario isn't that updated in last commit?

[Leilei332]

Would it be possible to seperate plugin language packs from plugin itself? IMO this can avoid unnecessary language to be installed and avoid the size of the tiddlywiki file be increased too much when the plugin has many languages.

[linonetwo]

Would it be possible to seperate plugin language packs from plugin itself?

@Leilei332 Yes. Mario already ask same question above.