locale_gen: de_CH.UTF-8 not being created on Ubuntu 24.04

[ltog]

Summary

I try to use the Ansible module locale_gen to create the locale de_CH.UTF-8 on Ubuntu 24.04 by this task:

- name: "Add de_CH.UTF-8 locale"
  community.general.locale_gen:
    name: de_CH.UTF-8
    state: present

The task returns OK on both the first and subsequent runs, however, every run takes approx. 23s on a local VM on modern hardware, indicating that compilation is taking place for each run. Worse, the outputs of all of the following commands lack the desired locale, implying that it was not successfully built:

• locale -a
• localedef --list-archive
• localectl list-locales

Expected result: The de_CH.UTF-8 locale showing up in the output of all of these commands.

For testing I use molecule in combination with https://portal.cloud.hashicorp.com/vagrant/discover/alvistack/ubuntu-24.04 on one hand and a custom virtual machine on the other hand. Both show the same symptoms.

From the module output I can see that ubuntu_mode is true. That makes sense since /var/lib/locales/supported.d/ is present on both virtual machines. Looking at the source it seems that in apply_change_ubuntu() either set_locale() should be called (as being done in apply_change) or the compilation of the locale should be triggered by a corresponding entry de_CH.UTF-8 UTF-8 in file /var/lib/locales/supported.d/local, neither is being done. To my understanding the comment in

community.general/plugins/modules/locale_gen.py

Line 217 in bafb8ac

# Ubuntu's patched locale-gen automatically adds the new locale to /var/lib/locales/supported.d/local

is wrong.

As a workaround I install the apt package language-pack-de which creates the desired locale, but results in the creation of other locales as well.

Issue Type

Bug Report

Component Name

locale_gen

Ansible Version

$ ansible --version
279753 1731683005.03029: starting run
ansible [core 2.17.6]
  config file = /home/XXXXXXXX/ansible-environment/ansible.cfg
  configured module search path = ['/home/XXXXXXXX/library']
  ansible python module location = /home/XXXXXXXX/venv-ansible/lib/python3.10/site-packages/ansible
  ansible collection location = /home/XXXXXXXX/collections
  executable location = /home/XXXXXXXX/venv-ansible/bin/ansible
  python version = 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0] (/home/XXXXXXXX/venv-ansible/bin/python3)
  jinja version = 3.1.4
  libyaml = True

Community.general Version

$ ansible-galaxy collection list community.general
Collection        Version
----------------- -------
community.general 9.5.1

Configuration

$ ansible-config dump --only-changed

OS / Environment

Target OS: Ubuntu 24.04.1 LTS

Steps to Reproduce

- name: "Add de_CH.UTF-8 locale"
  community.general.locale_gen:
    name: de_CH.UTF-8
    state: present

Expected Results

Locale is being generated.

Actual Results

Locale is not being generated.

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibullbot]

Files identified in the description:

• plugins/modules/locale_gen.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibullbot]

cc @AugustusKling
click here for bot help

[russoz]

Hi @ltog thanks for reporting.

I am trying to reproduce the error, but I do see the locale added when running in Ubuntu 24.04. The only thing I noticed is that the actual locale installed gets a slightly different name, it becomes de_CH.utf8.

There are many locales that are named differently when installed. To the best of my knowledge that is inherent to the locale system and that's exactly how it should work.

[ltog]

@russoz : Thanks for having a look! How did you check the installation of the locale?

[russoz]

@ltog I was adding an extra case to the test integrations. Running them with -vvv gave me:

...

TASK [locale_gen : Is the locale present? de_CH.UTF-8] *************************
task path: /tmp/andebox._imqqjr1/ansible_collections/community/general/tests/output/.tmp/integration/locale_gen-mw5gc1bb-ÅÑŚÌβŁÈ/tests/integration/targets/locale_gen/tasks/basic.yml:49
Using module file /tmp/ansible-test-3dh9_knp/ansible/modules/command.py
Pipelining is enabled.
<testhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<testhost> EXEC /bin/sh -c '/venv/bin/python && sleep 0'
changed: [testhost] => {
    "changed": true,
    "cmd": [
        "locale",
        "-a"
    ],
    "delta": "0:00:00.005859",
    "end": "2024-11-22 01:54:17.305826",
    "invocation": {
        "module_args": {
            "_raw_params": "locale -a",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "expand_argument_vars": true,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true
        }
    },
    "msg": "",
    "rc": 0,
    "start": "2024-11-22 01:54:17.299967",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "C\nC.utf8\nde_CH.utf8\nen_US.utf8\nPOSIX",
    "stdout_lines": [
        "C",
        "C.utf8",
        "de_CH.utf8",
        "en_US.utf8",
        "POSIX"
    ]
}

...

[russoz]

hi @ltog: just a friendly check, any chance you had time to take a look at this?

[ltog]

@russoz : Yeah, I had another look but unfortunately the result is still the same as described above. I know about the different names for the locale but neither is showing up with locale -a.

Would you mind checking whether ubuntu_mode is being reported as being used/performed by the module in your environment when running Ansible with -vv? This will be the case when directory /var/lib/locales/supported.d/ exists. The code takes a different path in that case. If it is the case, would you mind looking at the code of the module? I can't find the place where we store the information that the selected locale shall be created. (In non-Ubuntu mode it is stored in /etc/locale.gen.) I suspect it is plain missing. It might be needed to be written into /var/lib/locales/supported.d/local but I am unsure about that.

[russoz]

@ltog bingo! The /var/lib/locales/supported.d/ directory does not exist in the image I am using, thus I am getting ubuntu_mode=false. I am seeing comments around in forums describing images/containers that perform:

ln -s /usr/share/i18n/SUPPORTED /var/lib/locales/supported.d/all

while provisioning the machine/container.

I'll dig more on this.

[ltog]

@russoz : Great, we have a trail! Keep me posted.

[russoz]

Will do but, I got a string of busy days ahead of me, I am probably working on this again by the end of next week

[ltog]

@russoz : Ok, thanks for working on it!

[russoz]

Hi @ltog @lukasz-zaroda,

I did not have time for implementing a solution, but I managed to get the investigation going. Bottom line seems to be the fact that there are two different mechanisms to generate locales, the standard glibc one and the Debian one. The ubuntu_mode flag is set to True when /var/lib/locales/supported.d/ exists, and that directory is part of the Debian mechanism. Locale enabling/disabling happens by editing /var/lib/locales/supported.d/local. OTOH the glibc mechanism uses /etc/locale.gen to enable/disable.

Ubuntu has used both mechanisms for a long time, but starting in Ubuntu 16.04 they seem to have made a decision to let go of the Debian mechanism and use only glibc as the primary mechanism. I imagine/suppose that the tools still provide support for the Debian mechanism - but not sure, nor how long that's gonna last.

To the best of my knowledge, both mechanisms claim to take the list of available locales from /usr/share/i18n/SUPPORTED, but currently the module reads that when ubuntu_mode=true and it reads /etc/locale.gen itself (!?!?) when false.

My intended approach on this is to rewrite the module with the following logic:

1. If there is a file /etc/locale.gen then it will assume glibc mode (ubuntu_mode=false), making that mode the "primary/default/standard" mechanism - it should cover all standard Ubuntu images from 16.04 onwards
2. Else, it will look for /var/lib/locales/supported.d/ and, if found, will assume Debian mode (ubuntu_mode=true).
3. Else it will raise a failure.
4. Look into /usr/share/i18n/SUPPORTED for supported locales and enable/disable the requested locales in the file corresponding the mechanism being used.

That way we are going to have glibc mode as the default, covering virtually most of the cases, and Debian mode for Ubuntu backward compatibility or Debian compatibility (Debian still uses that up to this day, according to ChatGPT :-) - I will test that eventually but not a priority for now).

The output variables should not be affected by this - though later on I would want to rename the return value ubuntu_mode to something like debian_mode or something. Also in a later change, by having glibc mode as a default, it should be easy to enable that module for RHEL-based systems (not supported today as per the module's notes). The existing tests should pass, but they do not take ubuntu_mode into account, so new tests (or re-running everything ubuntu_mode=true) should be added.

So, I am planning on starting this work slowly in the next couple of days, to give you guys time to read, ponder and comment on it as you see fit. I reckon by the next weekend I should have time to wrap it up and submit the PR. Please let me know what you think.

Mentioning #8487 so anyone tracking that one gets notified.

Cc: @felixfontein (for your sharp eyes ;-) )

[russoz]

Sorry, missed @samikhelil83 on the comment above.

[russoz]

@ltog @lukasz-zaroda could you please give it a try with the code from the PR?

The code for ubuntu_mode=True only seems to work when /etc/locale.gen is present, in which case the module now uses mechanism=glibc (a.k.a. ubuntu_mode=False). If /etc/locale.gen is removed, locale-gen seems to do nothing at all. The tests running against Debian system seem to have picked up mechanism=glibc as well.

That points to removing ubuntu_mode=True. @felixfontein not sure how would the best way to handle this now - would we need to deprecate that mode? It does not seem to be working at all.

[felixfontein]

That points to removing ubuntu_mode=True. @felixfontein not sure how would the best way to handle this now - would we need to deprecate that mode? It does not seem to be working at all.

My guess is that it does work with very old versions of Ubuntu. Since someone might still use the module successfully with these, we should deprecate it.

[russoz]

OK, just a couple of comments:

1. I was unable to make it work in local tests - I think I tried back to Ubuntu 16.04 but the 14.04 failed to install something and I ended up letting go, so I cannot be sure it actually works.
2. Since the work around the deprecation is not in the module - but installing a new version of the OS - I think we should announce the deprecation, add that to the docs, add comments to the code, bang drums and blow the horns about it, but we should refrain from calling module.deprecate(), and set an extended time for removal.

Does that look good to you?

[ltog]

@russoz : Thanks for working on this. I did some tests.

• Ubuntu 20.04, 22.04 and 24.04 seem to work fine.
• I could not test 18.04 and previous version, to my understanding due to a too old Python version. I did not dig too deep there.
• I noticed that in --check mode the task will report ok, although it should be changed. Would it be possible to adjust that?

[russoz]

hi @ltog , thanks for that! :-)

Well, about the versions, my plan is to deprecate support to anything that is not in the CI pipeline, so that should be OK.

About the check mode, I would rather do that in a separate PR, because it would be a bugfix and those are backported to previous releases.