Translation - scripts, the final part #154

marmarek · 2021-03-31T18:37:27Z

This is the final part of the translation integration. Previous two:
#153
QubesOS/qubes-doc#1123

Similar to previous parts, the majority of work was done by @maiska and @tokideveloper. Thanks!

On top of their work, this PR integrates all those scripts with Gitlab CI pipeline. It handles pushing/pulling content to/from Transifex automatically. I have performed the first iteration of this, to populate https://github.com/QubesOS/qubes-translated repository with initial content (we did have some content translated before, so indeed some (small) parts are translated, mostly in Spanish language). For the test purposes, I've enabled languages es, de, fr, but this commit should not be merged until substantial part of the documentation there is translated (right now most of those pages are simply copies from English).

See README for high level description of how it works.

The resulting website, with translations enabled can be seen at https://wwwtest.qubes-os.org/

marmarek · 2021-03-31T22:08:30Z

testdeploy

tokideveloper · 2021-04-01T20:43:53Z

Great work! Thanks to @maiska and @marmarek!

BTW, the line

{% assign langmenu = (site.languages.size > 1) %}

doesn't seem to work as expected. At least, I see the lang menu with the English entry only. The following should work (without having tested it):

{% if site.languages.size > 1 %}
  {% assign langmenu = true %}
{% else %}
  {% assign langmenu = false %}
{% endif %}

It's a Liquid thing …

marmarek · 2021-04-01T20:54:23Z

{% assign langmenu = (site.languages.size > 1) %}

It works, just ignore wwwpreview instance - it got built with empty _translated dir... See wwwtest instance, linked in the PR description.

tokideveloper · 2021-04-01T21:35:09Z

{% assign langmenu = (site.languages.size > 1) %}

It works, just ignore wwwpreview instance - it got built with empty _translated dir... See wwwtest instance, linked in the PR description.

I see multiple langs in the menu in the wwwtest instance, so, it seems to work as you said.

However, I tried the following mini examples right now:

Example 1

{% assign a = (10 > 5) %}
{{ a }}

outputs nothing. Also, I got a syntax error warning.

Example 2

{% assign a = 10 > 5 %}
{{ a }}

outputs 10. Also, I got a syntax error warning.

Example 3

{% if 10 > 5 %}
  {% assign a = true %}
{% else %}
  {% assign a = false %}
{% endif %}
{{ a }}

outputs true. No warning.

Example 4

{% if (10 > 5) %}
  {% assign a = true %}
{% else %}
  {% assign a = false %}
{% endif %}
{{ a }}

outputs false. Also, I got a syntax error warning.

Do I have a different Liquid version?
What's the output of {{ langmenu }}?

Sorry for the noise. I just want to avoid errors in the long run.

marmarek · 2021-04-02T03:17:54Z

You are correct, it worked only because langmenu was assigned 4. Thanks!

tokideveloper · 2021-04-16T09:50:24Z

_utils/_translation_utils/post_transifex_pull.sh

+
+echo "============================ post processing step 2 ======================================"
+#read b
+ruby _utils/_translation_utils/merge_md_heading_ids.rb "$1" /tmp/tx-mapping


At least for the merge_md_heading_ids.rb script, called here, it is important that the checked out commit is not the most recent one but that commit that was the current one when the files have been pushed to Transifex the last time. The reason is that headlines may have changed or added or removed meanwhile and thus, the merge_md_heading_ids.rb script may produce a wrong output.

I can't see any attempt to checkout that correct commit or have I overseen something?

IMHO when pushing files to Transifex, the related commit hash should be saved. When pulling the files from Transifex, that related (old) commit should be checked out for the postprocessing steps.

Is it understandable what I mean?

This all is called just after pushing and pulling from transifex. See https://github.com/QubesOS/qubesos.github.io/pull/154/files#diff-037ea159eb0a7cb0ac23b851e66bee30fb838ee8d0d99fa331a1ba65283d37f7R38-R39

Ah, okay, now I see. Both pushing and pulling belong to the same commit and thus everything is right. Thank you for clarification!

marmarek · 2021-06-23T02:47:38Z

I've rebased it on top of recent version and added few improvement discussed in the meantime (including @tokideveloper 's updated script).
This PR includes also #177

_includes/doc-widget.html

_includes/header.html

_includes/news-post-author.html

_includes/news.html

_includes/team-heading.html

_includes/team-simple.html

marmarek · 2021-06-23T13:19:18Z

I'll simplify it further when rebasing on top of #178 (when it will be merged).

marmarek · 2021-06-24T04:23:51Z

PipelineRefresh

marmarek · 2021-06-24T12:15:17Z

PipelineRetry

marmarek · 2021-06-24T14:16:40Z

PipelineRetry

marmarek · 2021-06-24T15:52:27Z

PipelineRetry

Remove translated files when the source file get removed. This includes also renaming files.

Do not attempt to verify correctness of the website before refreshing all languages. If any structural change happened (rename, permalink change etc), it will fail because of the old files, not the freshly downloaded ones. Make htmlproofer postprocessing script language agnostic. This is mostly about removing "language" parameter, which wasn't used anyway.

andrewdavidwong · 2021-06-24T16:33:19Z

Just to confirm... you're not waiting or expecting me to do anything with this, right @marmarek? I've been intentionally leaving this PR alone because you seem like you're managing it yourself (and of course the big "[DO NOT MERGE]" warning).

marmarek · 2021-06-24T17:30:03Z

testdeploy

marmarek · 2021-06-24T17:32:26Z

trestdeploy

marmarek · 2021-06-24T17:32:32Z

testdeploy

marmarek · 2021-06-24T17:35:06Z

You are correct. I want to get it working again first. It's almost there. For example QubesOS/qubes-translated@a102a9e was just made. There is still some issue with language switcher (just "en" visible there).

As for the "[DO NOT MERGE]" commits - it's just about that commit - it enables the translations to be visible for some languages (es, fr, de) - which I think we want only when sufficient part will be translated already.

tokideveloper · 2021-06-24T20:07:20Z

You are correct. I want to get it working again first. It's almost there. For example QubesOS/qubes-translated@a102a9e was just made. There is still some issue with language switcher (just "en" visible there).

Currently (if language switcher enabled at all), for a certain page P, the language switcher only shows those languages where translations exist for P in qubes-translated. Should it be different?

marmarek · 2021-06-24T20:21:45Z

Currently (if language switcher enabled at all), for a certain page P, the language switcher only shows those languages where translations exist for P in qubes-translated.

I think this design is fine (even if may look weird at places). The issue is I see only "en" on pages I manually verified the "translated" version exists in the qubes-translated repo. For example documentation index, or developer/building/qubes-builder.md.

tokideveloper · 2021-06-24T20:52:58Z

Currently (if language switcher enabled at all), for a certain page P, the language switcher only shows those languages where translations exist for P in qubes-translated.

I think this design is fine (even if may look weird at places). The issue is I see only "en" on pages I manually verified the "translated" version exists in the qubes-translated repo. For example documentation index, or developer/building/qubes-builder.md.

That's strange because I tested it on my machine a few minutes ago and it works. Okay, actually I only tested two files: _doc/doc.md and _doc/user/how-to-guides/how-to-copy-and-paste-text.md. And actually, I just copied the two original files from qubesos.github.io/_doc to the newly created _qubes-translated directory (with underscore), set lang and permalink appropriately and translated title test-wise. That was sufficient to work: On these two pages, the switcher contains both en and de, on the other pages not.

EDIT: Of course, I enabled the switcher in _includes/header.html via {% assign langmenu = true %} in line 14.

marmarek · 2021-06-24T21:42:02Z

testdeploy

marmarek · 2021-06-24T21:43:32Z

Ok, that was not very smart from me. "testdeploy" command took the version without downloaded translations 🤦 Now it is better :)

tokideveloper · 2021-06-24T22:09:39Z

Now it is better :)

It's a lot better! It looks great! 🎉 ❤️

marmarek · 2021-06-25T01:22:27Z

https://wwwpreview.qubes-os.org/de/team/#inquiries - this doesn't look properly handled, especially "Business" should be "Geschäft" (it is translated in the qubes-translated repo). This is because _includes/team.html has:

  {% for team in site.data.team %}
    {% if team.type == "inquiries" %}
...

I can easily change that to appropriate entry from site.translated, but then team.type may not match "inquiries" if it gets translated. And in fact team.type core is translated to Kern in German version. I think team.type entry should not be translated, as it's just an internal metadata for the layout - the actual section titles are in team-page.yml file.

Some ideas:

mark team.type entry as non translatable (locked?)
don't translate team.yml file at all, but move translatable strings into team-page.yml ("name" in the inquiries section, "role" in others)

I'm not sure how to do either of those.

marmarek · 2021-06-25T01:29:59Z

Anyway, now is the time to review both the content of this very PR, but more importantly its output at wwwpreview.

As for the scripts in _utils/_translation_utils, some extra review would be useful, but not strictly necessary - those were mostly written by @maiska and @tokideveloper and already reviewed (and sometimes modified) by me.

tokideveloper · 2021-06-25T08:22:45Z

https://wwwpreview.qubes-os.org/de/team/#inquiries - this doesn't look properly handled, especially "Business" should be "Geschäft" (it is translated in the qubes-translated repo). This is because _includes/team.html has:
  {% for team in site.data.team %}
    {% if team.type == "inquiries" %}
...
I can easily change that to appropriate entry from site.translated, but then team.type may not match "inquiries" if it gets translated. And in fact team.type core is translated to Kern in German version. I think team.type entry should not be translated, as it's just an internal metadata for the layout - the actual section titles are in team-page.yml file.

Some ideas:

mark team.type entry as non translatable (locked?)

Yes! Maybe other entries need to be locked, too.

don't translate team.yml file at all, but move translatable strings into team-page.yml ("name" in the inquiries section, "role" in others)

No. I like the current design where the team itself has its own data file.

I'm not sure how to do either of those.

Me neither. @maiska, can you describe how to lock certain strings? Maybe via the API?

tokideveloper · 2021-06-25T08:39:55Z

Anyway, now is the time to review both the content of this very PR, but more importantly its output at wwwpreview.

I agree. First, the output should be good. Afterwards, we can review the code.

As for the scripts in _utils/_translation_utils, some extra review would be useful, but not strictly necessary - those were mostly written by @maiska and @tokideveloper and already reviewed (and sometimes modified) by me.

In the past, I reviewed maiska's files and they seemed to be good. But I can do another review again.

tokideveloper · 2021-06-25T11:12:53Z

mark team.type entry as non translatable (locked?)

Yes! Maybe other entries need to be locked, too.

The smart tag "locked" means "nobody can touch it" and the smart tag "notranslate" means in our case "replace with source string". We should use both. See https://docs.transifex.com/projects/preventing-resource-edits#locking-strings

Me neither. @maiska, can you describe how to lock certain strings? Maybe via the API?

See _utils/_translation_utils/tag_strings_as_locked.py. You may need to load the diff manually.

tokideveloper

I've reviewed all files. At least, I tried it and tried to understand the files.

tokideveloper · 2021-06-27T15:53:05Z

_utils/_translation_utils/COUNTER.txt

@@ -0,0 +1 @@
+current counter: 251


Can we be sure that this value is correct once the PR is merged?

Yes, I'll rerun the script shortly after merging again. And generally, once it lands in master, the script will be called frequently.

tokideveloper · 2021-06-28T15:17:07Z

_config.yml

+- de
+- fr
+- es
+


I would start with en only for now. This way, we could test the workflow on the live system.

That's the plan. Adding those extra languages is a part of the "[DO NOT MERGE] enable languages: de fr es" commit.

tokideveloper · 2021-06-28T15:25:23Z

_utils/_translation_utils/merge_md_heading_ids.py

+            result = merge_ids_in_gfm_files(original_lines, translated_lines)
+            write_lines(result, item)
+
+


Actually, this Python script isn't needed anymore since it's not called and it has been ported to a Ruby script (which is called). See _utils/_translation_utils/merge_md_heading_ids.rb.

EDIT: Also, this Python version is not up-to-date.

Ah, right. I'll remove it.

tokideveloper · 2021-06-28T16:14:29Z

_utils/_translation_utils/merge_md_heading_ids.rb

+        # headline starting with '#'
+        gfm_lines[line_number] = '# ' + placeholder + "\n"
+        hl = look_for_headline(render(gfm_lines), placeholder)
+    elsif next_line.length >= 3 and (line_only_made_of(next_line, '=') or line_only_made_of(next_line, '-'))


Since next_line contains a \n at the end, the check above must be next_line.length >= 3 + 1 instead of next_line.length >= 3.

Good point.

tokideveloper · 2021-06-28T16:45:35Z

_utils/_translation_utils/merge_md_heading_ids.rb

+
+    mapping = create_dict_from_tx_config(ARGV[0], ARGV[1])
+    mapping.each do |key, value|
+        if !key.end_with?(".yml")


Since there could be more files on Transifex in the future (e.g. HTML, PO etc), the check should check for accepted file endings like *.md: if key.end_with?(".md")

Good point.

tokideveloper · 2021-06-29T12:32:33Z

What I forget in the review: Some scripts run recursively through directories like _doc but don't ignore .git. To be corrected.

tokideveloper

In this review, I marked the spots where .git is not ignored.

Sometimes there seems to be no need to ignore .git since the root directory of such a walk is a repo subdirectory (not containing any .git directory). However, to be on the safe side, .git directories should be ignored in general.

tokideveloper · 2021-06-29T16:23:06Z

_utils/_translation_utils/postprocess_htmlproofer.py

+    mapping = {}
+    perms = []
+    yml_files = []
+    for dirname, subdirlist, filelist in walk(translated_dir):


This walk doesn't ignore .git.

tokideveloper · 2021-06-29T16:28:28Z

_utils/_translation_utils/postprocess_translation.py

+
+    href = set()
+    reg ='(?<=href=\").*?(?=\")'
+    for dirname, subdirlist, filelist in walk(translated_dir):


This walk doesn't ignore .git.

tokideveloper · 2021-06-29T16:28:56Z

_utils/_translation_utils/postprocess_translation.py

+    """
+
+    perms = set()
+    for dirname, subdirlist, filelist in walk(translated_dir):


This walk doesn't ignore .git.

tokideveloper · 2021-06-29T16:30:27Z

_utils/_translation_utils/prepare_for_translation.py

+
+        return counter
+
+    for dir_name, subdir_list, file_list in os.walk(root_dir):


This walk doesn't ignore .git.

jermanuts · 2024-07-15T08:13:05Z

Any progress?

tokideveloper · 2024-07-16T09:57:02Z

Any progress?

This PR focuses on internationalization of the Qubes OS website including the documentation, using Markdown and Jekyll.

Meanwhile, we moved away from that approach towards a conversion of the documentation from Markdown to reStructuredText, render it using Sphinx and host it on ReadTheDocs. This is our (new) next step.

See QubesOS/qubes-doc#1367

The next steps afterwards would be internationalization and localization of the documentation and later the Qubes OS website.

marmarek force-pushed the translation-scripts branch from e4b2480 to 524b89a Compare March 31, 2021 18:53

qubesos-bot temporarily deployed to qa March 31, 2021 22:08 Inactive

marmarek force-pushed the translation-scripts branch from 524b89a to 680545b Compare April 2, 2021 03:20

tokideveloper reviewed Apr 16, 2021

View reviewed changes

andrewdavidwong assigned marmarek Jun 8, 2021

marmarek force-pushed the translation-scripts branch from 680545b to bde5342 Compare June 23, 2021 01:56

marmarek force-pushed the translation-scripts branch 2 times, most recently from cfdcf4e to e8d7029 Compare June 23, 2021 03:47