Language Selector :)

[sneakers-the-rat]

Wanted to help out the translation effort, but since i am a reprehensible monolingual don't have much to add.

Here's a draft of adding a language selector and making the translated builds a more regular part of the build process (existing stuff in the noxfile was already nicely done!)

language_selector.mp4

some notes

I put the "smarts" about locations into the templating process rather than in the javascript. this gives us some hope of being reusable in some way (maybe) since you have to know about the subdirectory you're in on the server somehow, and afaik the only good way to know that is from the base_url, which isn't available in the js (i could have embedded it in some empty HEAD element or something but i sorta like this better). I think that screenreaders don't read value but rather the label, but i will double check.

anyway that lets the js be nice and simple :)

it looks like the precommit action did something funky to my jinja templates (L3-9 in language_selector.html ) splitting it out across multiple lines. not sure if that's valid, so may have to figure out how to ignore that line in precommit or w/e.

I couldn't find a good way to fork a process or run concurrent processes with nox ( wntrblm/nox#198 (comment) ) so I just used concurrently in the global env... not the best so lmk if y'all know of a better way (probably python would be good so it can be installed by nox?). anyway it builds all the languages in parallel in dev mode and doesn't make stdout a nightmare by labeling them:

single-sourced the languages from the conf.py file, hopefully thats ok, wanted to make it easy to add new languages.

didn't do styles yet, wanted to see if this general approach seemed ok before i went hogwild.

[sneakers-the-rat]

I'm also not sure if there's a standards-compliant way to signal that there are multiple languages available for a given page at different URLs, but if so we should do that.

edit: oop i guess you can't declare env variables like that in nox. it works in dev mode with nox -s docs-live-langs. will fix in the morning

[tkoyama010]

This is awesome!!! Thank you for trying this 👏

[flpm]

This is awesome and I think it is exactly what we were thinking about! 😍

The code for the Japanese language is ja instead of jp. I had made the same assumption, but @tkoyama010 sorted it out :) When I made that change in conf.py I saw for the first time the cool looking Japanese homepage 😍 I had not seen it yet.

About the language lists and how the selector is populated, here are some clarifications from the nox file perspective:

• The LANGUAGES list contains the languages for which nox will build and maintain locales (.po files). It is not what is used to build. I do not see any issue with having it in conf.py, it actually makes sense as it is a configuration option.
• The RELEASE_LANGUAGES is what nox will use to build for release to the site (when nox -s docs-test is called in CI). It should not be automatically all languages minus English because we might want to have enough translated (@lwasser had suggested 50%). It could also be a list in conf.py.
• I think the challenge is which one should the selector use to populate its items. Could it do with different lists depending on the 'dev' versus 'release' scenario? If not, it should then follow the RELEASE_LANGUAGES list, because otherwise we would have a drop down option in the site that points to a translation that has not been publish yet (404)

[willingc]

Love, love, love this @sneakers-the-rat. Thank you!

[sneakers-the-rat]

aha, so yes some of these problems are from the special-casing of en, and i think there needs to be some tidying up around how we're specifying a dev/prod build since there are currently multiple ways to specify that

• I introduced a SPHINX_DEV environment variable to switch the baseurl, since the deployed docs are served from a subdirectory python-package-guide/ rather than from / as in dev
• Dev/prod is also declared by running different nox sessions, *_test vs *
• there is posarg for release-build in the build_translations as well.

Would it be alright if we just followed node/rails and did like SPHINX_ENV=development or SPHINX_ENV=production ?

Then we can switch which languages are included that way and be relatively sure of it.

The RELEASE_LANGUAGES is what nox will use to build for release to the site (when nox -s docs-test is called in CI). It should not be automatically all languages minus English because we might want to have enough translated

restoring this now, i saw that previously but them being empty was making me sad lol. I don't have strong opinions at all, but maybe another thing we could do is publish all started translations and then on pages that haven't been translated add a message like "this isn't translated yet, but you can help us translate!" with a link to docs or to the relevant .po file?

I think the challenge is which one should the selector use to populate its items.

this is one of the reasons i wanted to single-source this. sphinx should use the same list that nox is using when building so these don't get out of sync, i'll clean this up a bit now

[sneakers-the-rat]

OK I updated so

• language is correct for Japanese (sorry!)
• from within sphinx, all behavior is switched by SPHINX_ENV, either development or production. I also tried to unify this in the noxfile, where some sessions have the env var forced (eg. autobuilds are only development, docs-test is only prod) and documented in README. To do that i made a little helper function that first checks for one of SPHINX_ENVS in the positional args, and if none is present reading from os.environ, defaulting to development. It looks like it works to me but check my work.

The main reason why this is such a pain is that 1) sphinx doesn't handle having a base_dir well like webpack, jekyll, and other ssgs do, and 2) en is special cased to not break already-existing inbound links - if all urls had some language code in them, then we could just do a regex substitution in the current page url in js. since we don't, we have to be able to parse where we are currently and know where to either add or strip the language code in the url. since we could either be at the root of the domain (as in dev) or under some subdirectory (as in prod), we need some way of forwarding that through to the built page. atm i am just hardcoding that in the value of each selection for each page using the template, but there is probably a better way.

the other source of complexity is that normally language building is switched like sphinx-build -D language={lang}, but that happens after the execution of conf.py, so we also need to pass SPHINX_LANG so that the context passed to the template knows about the current language. (technically the -D based language switching is unnecessary with the env var, but let's keep it for convention). If we were to cook up a better way of resolving the inter-language links then we could remove that too

[flpm]

• there is posarg for release-build in the build_translations as well.

Yeah that was a way for the docs-test to indicate to the build-translations session that it should only builds the RELEASE_LANGUAGES. Calling the build-translations directly (without that argument) will build all LANGUAGES.

Would it be alright if we just followed node/rails and did like SPHINX_ENV=development or SPHINX_ENV=production ?

Sure, I think that is fine

restoring this now, i saw that previously but them being empty was making me sad lol. I don't have strong opinions at all, but maybe another thing we could do is publish all started translations and then on pages that haven't been translated add a message like "this isn't translated yet, but you can help us translate!" with a link to docs or to the relevant .po file?

It's a good idea but I am not sure how to do that with Sphinx. The .po files are not one to one with the original files, it's more one to one with the sections.

this is one of the reasons i wanted to single-source this. sphinx should use the same list that nox is using when building so these don't get out of sync, i'll clean this up a bit now

👍

[sneakers-the-rat]

Yeah that was a way for the docs-test to indicate to the build-translations session that it should only builds the RELEASE_LANGUAGES. Calling the build-translations directly (without that argument) will build all LANGUAGES.

I saw this and preserved it :) now the same behavior can be done by passing "production" (or "development") as a posarg, which overrides the env var. Learned a bit about how nox works from this. Nifty little tool even if I still want to figure out how to only make one venv instead of 10 ♥

It's a good idea but I am not sure how to do that with Sphinx. The .po files are not one to one with the original files, it's more one to one with the sections.

I found a config option that adds CSS classes to blocks/divs that indicates their translated status: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-translation_progress_classes

So I think we can make this visible in-page just with CSS classes, and perhaps add some JS to add a little banner at the top if there are untranslated changes to a page. There is also a substitution (hadn't ever used a docutils substitution before) that prints a percentage like

:::{eval-rst}
|translation progress|
:::

though I dont know how useful that would be unless I can style it with JS/put it in the sidebar/not in main text.

You're right tho I think linking out from an untranslated section to the relevant lInes in the .po file might be too tricky to be worth it, but we could link out to the whole .po file relatively easily by just parsing the URL I think?

Normally I would say lets save these for another PR but since this PR would make translations "live" on the site it might be worth getting all the things we would want roughly in place here, or else collect a few PRs in a translation branch and then merge them back to main once they're all done. Open to whatever, just trying to make the good work others have done visible!