From 03d8165bd142e34abf075892819cd33c0c6360e5 Mon Sep 17 00:00:00 2001 From: Charles Brunet Date: Thu, 7 Nov 2024 10:58:14 -0500 Subject: [PATCH] New find_xgettext method to i18n module This method return a XgettextProgram object used to extract translatable string from source files into a .pot translation template. It differs from a plain CustomTarget in three ways: - It accepts build targets as sources, and automatically resolves source files from those build targets; - It detects command lines that are too long, and writes, at config time, the list of source files into a text file to be consumed by the xgettext command; - It detects dependencies between pot extraction targets, based on the dependencies between source targets. --- docs/markdown/i18n-module.md | 67 ++++++ docs/markdown/snippets/i18n_extract.md | 13 ++ mesonbuild/modules/i18n.py | 200 +++++++++++++++++- .../38 gettext extractor/meson.build | 18 ++ .../38 gettext extractor/src/lib1/lib1.c | 10 + .../38 gettext extractor/src/lib1/lib1.h | 6 + .../38 gettext extractor/src/lib1/meson.build | 3 + .../38 gettext extractor/src/lib2/lib2.c | 13 ++ .../38 gettext extractor/src/lib2/lib2.h | 6 + .../38 gettext extractor/src/lib2/meson.build | 3 + .../38 gettext extractor/src/main.c | 8 + .../38 gettext extractor/src/meson.build | 6 + .../frameworks/38 gettext extractor/test.json | 6 + 13 files changed, 355 insertions(+), 4 deletions(-) create mode 100644 docs/markdown/snippets/i18n_extract.md create mode 100644 test cases/frameworks/38 gettext extractor/meson.build create mode 100644 test cases/frameworks/38 gettext extractor/src/lib1/lib1.c create mode 100644 test cases/frameworks/38 gettext extractor/src/lib1/lib1.h create mode 100644 test cases/frameworks/38 gettext extractor/src/lib1/meson.build create mode 100644 test cases/frameworks/38 gettext extractor/src/lib2/lib2.c create mode 100644 test cases/frameworks/38 gettext extractor/src/lib2/lib2.h create mode 100644 test cases/frameworks/38 gettext extractor/src/lib2/meson.build create mode 100644 test cases/frameworks/38 gettext extractor/src/main.c create mode 100644 test cases/frameworks/38 gettext extractor/src/meson.build create mode 100644 test cases/frameworks/38 gettext extractor/test.json diff --git a/docs/markdown/i18n-module.md b/docs/markdown/i18n-module.md index a939a34738b5..66089822835a 100644 --- a/docs/markdown/i18n-module.md +++ b/docs/markdown/i18n-module.md @@ -74,3 +74,70 @@ for normal keywords. In addition it accepts these keywords: * `mo_targets` *required*: mo file generation targets as returned by `i18n.gettext()`. *Added 0.62.0* + + +### i18n.find_xgettext() + +``` meson +i18n.find_xgettext(args: [...], required: true) +``` + +Find the `xgettext` program, and return a `XgettextProgram` object. +`args` are a list of arguments to provide to `xgettext`. +If `required` is `true` and `xgettext` cannot be found, it will result in +an error. + +This function is to be used when the `gettext` function workflow it not suitable +for your project. For example, it can be used to produce separate `.pot` files +for each executable. + +*Added 1.7.0* + +#### `XgettextProgram` methods + +`XgettextProgram` is an [[@external_program]]. Therefore, it has the same methods, +and can be use in any places requiring an `ExternalProgram`. Furthermore, it has +the following method: + +##### `extract()` + +Positional arguments are the following: + +- name `str`: the name of the resulting pot file. +- sources `list[str|File|build_tgt|custom_tgt]`: + source files or targets. May be a list of `string`, `File`, [[@build_tgt]], + or [[@custom_tgt]] returned from other calls to this function. + +Keyword arguments are the following: + +- merge `bool`: + if `true`, will merge the resulting pot file with extracted pot files + related to dependencies of the given source targets. For instance, + if you build an executable, then you may want to merge the executable + translations with the translations from the dependent libraries. +- install `bool`: if `true`, will add the resulting pot file to install targets. +- install_tag `str`: install tag to use for the install target. +- install_dir `str`: directory where to install the resulting pot file. +- alias `list[build_tgt]`: + a list of build targets using the same resulting pot file. For + instance, if you build both a static and a shared library from the + same sources, you may specify the static library target as an alias, + to use the generated pot file when that static library is used as a + dependency of another target. + +The `extract()` method returns a [[@custom_tgt]]. + + +Usually, you want to pass one build target as sources, and the list of header files +for that target. If the number of source files would result in a command line that +is too long, the list of source files is written to a file at config time, to be +used as input for the `xgettext` program. + +The `merge: true` argument is to be given to targets that will actually read +the resulting `.mo` file. Each time you call the `extract()` method, it maps the +source targets to the resulting pot file. When `merge: true` is given, all +generated pot files from dependencies of the source targets are included to +generate the final pot file. Therefore, adding a dependency to source target +will automatically add the translations of that dependency to the needed +translations for that source target. + diff --git a/docs/markdown/snippets/i18n_extract.md b/docs/markdown/snippets/i18n_extract.md new file mode 100644 index 000000000000..beba96e9510d --- /dev/null +++ b/docs/markdown/snippets/i18n_extract.md @@ -0,0 +1,13 @@ +## i18n module find_xgettext + +There is a new `find_xgettext` function in `i18n` module that acts as a +wrapper around `xgettext`. It allows to extract strings to translate from +source files. + +This function is convenient, because: +- It can find the sources files from a build target; +- It will use an intermediate file when the number of source files is too + big to be handled directly from the command line; +- It is able to get strings to translate from the dependencies of the given + targets. + diff --git a/mesonbuild/modules/i18n.py b/mesonbuild/modules/i18n.py index 551e0b36fab6..f3d3168469ad 100644 --- a/mesonbuild/modules/i18n.py +++ b/mesonbuild/modules/i18n.py @@ -4,6 +4,8 @@ from __future__ import annotations from os import path +from pathlib import Path +import itertools import shlex import typing as T @@ -13,8 +15,10 @@ from ..options import OptionKey from .. import mlog from ..interpreter.type_checking import CT_BUILD_BY_DEFAULT, CT_INPUT_KW, INSTALL_TAG_KW, OUTPUT_KW, INSTALL_DIR_KW, INSTALL_KW, NoneType, in_set_validator -from ..interpreterbase import FeatureNew, InvalidArguments -from ..interpreterbase.decorators import ContainerTypeInfo, KwargInfo, noPosargs, typed_kwargs, typed_pos_args +from ..interpreter.interpreterobjects import _ExternalProgramHolder +from ..interpreterbase import FeatureNew +from ..interpreterbase.exceptions import InterpreterException, InvalidArguments +from ..interpreterbase.decorators import ContainerTypeInfo, KwargInfo, noPosargs, noKwargs, typed_kwargs, typed_pos_args from ..programs import ExternalProgram from ..scripts.gettext import read_linguas @@ -24,7 +28,7 @@ from . import ModuleState from ..build import Target from ..interpreter import Interpreter - from ..interpreterbase import TYPE_var + from ..interpreterbase import TYPE_var, TYPE_kwargs class MergeFile(TypedDict): @@ -65,6 +69,21 @@ class ItsJoinFile(TypedDict): its_files: T.List[str] mo_targets: T.List[T.Union[build.BuildTarget, build.CustomTarget, build.CustomTargetIndex]] + class XgettextProgramT(TypedDict): + + args: T.List[str] + required: bool + + class XgettextProgramExtract(TypedDict): + + merge: bool + install: bool + install_dir: T.Optional[str] + install_tag: T.Optional[str] + alias: T.List[T.Union[build.BuildTarget, build.BothLibraries]] + + SourcesType = T.Union[str, mesonlib.File, build.BuildTarget, build.BothLibraries, build.CustomTarget] + _ARGS: KwargInfo[T.List[str]] = KwargInfo( 'args', @@ -115,6 +134,161 @@ class ItsJoinFile(TypedDict): } +class XgettextProgram(ExternalProgram): + + def __init__(self, xgettext: ExternalProgram, args: T.List[str]): + self.__dict__.update(xgettext.__dict__) + self.command.extend(args) + + self.pot_files: T.Dict[str, build.CustomTarget] = {} + + def extract(self, + name: str, + sources: T.List[SourcesType], + merge: bool, + install: bool, + install_dir: T.Optional[str], + install_tag: T.Optional[str], + alias: T.List[T.Union[build.BuildTarget, build.BothLibraries]], + interpreter: Interpreter) -> build.CustomTarget: + + if not name.endswith('.pot'): + name += '.pot' + + source_files = self._get_source_files(sources, interpreter) + + command = self.command.copy() + command.append(f'--directory={interpreter.environment.get_source_dir()}') + command.append(f'--directory={interpreter.environment.get_build_dir()}') + command.append('--output=@OUTPUT@') + + depends = list(self._get_depends(sources)) if merge else [] + rsp_file = self._get_rsp_file(name, source_files, depends, command, interpreter) + inputs: T.List[T.Union[mesonlib.File, build.CustomTarget]] + if rsp_file: + inputs = [rsp_file] + depend_files = list(source_files) + command.append('--files-from=@INPUT@') + else: + inputs = list(source_files) + depends + depends = None + depend_files = None + command.append('@INPUT@') + + ct = build.CustomTarget( + '', + interpreter.subdir, + interpreter.subproject, + interpreter.environment, + command, + inputs, + [name], + depend_files = depend_files, + extra_depends = depends, + install = install, + install_dir = [install_dir] if install_dir else None, + install_tag = [install_tag] if install_tag else None, + description = 'Extracting translations to {}', + ) + + for source_id in self._get_source_id(itertools.chain(sources, alias)): + self.pot_files[source_id] = ct + self.pot_files[ct.get_id()] = ct + + interpreter.add_target(ct.name, ct) + return ct + + def _get_source_files(self, sources: T.Iterable[SourcesType], interpreter: Interpreter) -> T.Set[mesonlib.File]: + source_files = set() + for source in sources: + if isinstance(source, mesonlib.File): + source_files.add(source) + elif isinstance(source, str): + mesonlib.check_direntry_issues(source) + source_files.add(mesonlib.File.from_source_file(interpreter.source_root, interpreter.subdir, source)) + elif isinstance(source, build.BuildTarget): + source_files.update(source.get_sources()) + elif isinstance(source, build.BothLibraries): + source_files.update(source.get('shared').get_sources()) + return source_files + + def _get_depends(self, sources: T.Iterable[SourcesType]) -> T.Set[build.CustomTarget]: + depends = set() + for source in sources: + if isinstance(source, build.BuildTarget): + for source_id in self._get_source_id(source.get_dependencies()): + if source_id in self.pot_files: + depends.add(self.pot_files[source_id]) + elif isinstance(source, build.CustomTarget): + # Dependency on another extracted pot file + source_id = source.get_id() + if source_id in self.pot_files: + depends.add(self.pot_files[source_id]) + return depends + + def _get_rsp_file(self, + name: str, + source_files: T.Iterable[mesonlib.File], + depends: T.Iterable[build.CustomTarget], + arguments: T.List[str], + interpreter: Interpreter) -> T.Optional[mesonlib.File]: + source_list = '\n'.join(source.relative_name() for source in source_files) + for dep in depends: + source_list += '\n' + path.join(dep.subdir, dep.get_filename()) + + estimated_cmdline_length = len(source_list) + len(arguments) + sum(len(arg) for arg in arguments) + 1 + if estimated_cmdline_length < 8000: + # Maximum command line length on Windows is 8191 + # Limit on other OS is higher, but a too long command line wouldn't + # be practical in any ways. + return None + + rsp_file = Path(interpreter.environment.build_dir, interpreter.subdir, name+'.rsp') + rsp_file.write_text(source_list, encoding='utf-8') + + return mesonlib.File.from_built_file(interpreter.subdir, rsp_file.name) + + def _get_source_id(self, sources: T.Iterable[T.Union[SourcesType, build.CustomTargetIndex]]) -> T.Iterable[str]: + for source in sources: + if isinstance(source, build.Target): + yield source.get_id() + elif isinstance(source, build.BothLibraries): + yield source.get('static').get_id() + yield source.get('shared').get_id() + + +class XgettextProgramHolder(_ExternalProgramHolder[XgettextProgram]): + + def __init__(self, xgettext_program: XgettextProgram, interpreter: Interpreter) -> None: + super().__init__(xgettext_program, interpreter) + self.methods.update({ + 'extract': self.extract_method, + }) + + @typed_pos_args('XgettextProgram.extract', str, varargs=(str, mesonlib.File, build.BuildTarget, build.BothLibraries, build.CustomTarget), min_varargs=1) + @typed_kwargs( + 'XgettextProgram.extract', + KwargInfo('merge', bool, default=False), + KwargInfo('alias', ContainerTypeInfo(list, (build.BuildTarget, build.BothLibraries)), listify=True, default=[]), + INSTALL_KW, + INSTALL_DIR_KW, + INSTALL_TAG_KW, + ) + def extract_method(self, args: T.Tuple[str, T.List[SourcesType]], kwargs: XgettextProgramExtract) -> build.CustomTarget: + if kwargs['install'] and not kwargs['install_dir']: + raise InvalidArguments('XgettextProgram.extract: "install_dir" keyword argument must be set when "install" is true.') + + if not self.held_object.found(): + raise InterpreterException('XgettextProgram.extract: "xgettext" command not found.') + + return self.held_object.extract(*args, **kwargs, interpreter=self.interpreter) + + @noPosargs + @noKwargs + def found_method(self, args: TYPE_var, kwargs: TYPE_kwargs) -> bool: + return self.held_object.found() + + class I18nModule(ExtensionModule): INFO = ModuleInfo('i18n') @@ -125,6 +299,7 @@ def __init__(self, interpreter: 'Interpreter'): 'merge_file': self.merge_file, 'gettext': self.gettext, 'itstool_join': self.itstool_join, + 'find_xgettext': self.find_xgettext, }) self.tools: T.Dict[str, T.Optional[T.Union[ExternalProgram, build.Executable]]] = { 'itstool': None, @@ -398,6 +573,23 @@ def itstool_join(self, state: 'ModuleState', args: T.List['TYPE_var'], kwargs: ' return ModuleReturnValue(ct, [ct]) + @FeatureNew('i18n.find_xgettext', '1.7.0') + @noPosargs + @typed_kwargs( + 'i18n.find_xgettext', + _ARGS, + KwargInfo('required', bool, default=True), + ) + def find_xgettext(self, state: ModuleState, args: TYPE_var, kwargs: XgettextProgramT) -> XgettextProgram: + tool = 'xgettext' + if self.tools[tool] is None or not self.tools[tool].found(): + self.tools[tool] = state.find_program(tool, required=kwargs['required'], for_machine=mesonlib.MachineChoice.BUILD) + + xgettext = T.cast(ExternalProgram, self.tools[tool]) + return XgettextProgram(xgettext, kwargs['args']) + def initialize(interp: 'Interpreter') -> I18nModule: - return I18nModule(interp) + mod = I18nModule(interp) + mod.interpreter.append_holder_map(XgettextProgram, XgettextProgramHolder) + return mod diff --git a/test cases/frameworks/38 gettext extractor/meson.build b/test cases/frameworks/38 gettext extractor/meson.build new file mode 100644 index 000000000000..dd4c818a3d08 --- /dev/null +++ b/test cases/frameworks/38 gettext extractor/meson.build @@ -0,0 +1,18 @@ +project( + 'gettext extractor', + 'c', + default_options: {'default_library': 'static'}, + meson_version: '1.7.0', +) + +i18n = import('i18n') +xgettext = i18n.find_xgettext( + args: ['-ktr', '--add-comments=TRANSLATOR:', '--from-code=UTF-8'], + required: false, +) + +if not xgettext.found() + error('MESON_SKIP_TEST xgettext command not found') +endif + +subdir('src') diff --git a/test cases/frameworks/38 gettext extractor/src/lib1/lib1.c b/test cases/frameworks/38 gettext extractor/src/lib1/lib1.c new file mode 100644 index 000000000000..723edda00637 --- /dev/null +++ b/test cases/frameworks/38 gettext extractor/src/lib1/lib1.c @@ -0,0 +1,10 @@ +#include "lib1.h" + +#include + +#define tr(STRING) (STRING) + +void say_something(void) +{ + printf("%s\n", tr("Something!")); +} diff --git a/test cases/frameworks/38 gettext extractor/src/lib1/lib1.h b/test cases/frameworks/38 gettext extractor/src/lib1/lib1.h new file mode 100644 index 000000000000..6199d29c4ec6 --- /dev/null +++ b/test cases/frameworks/38 gettext extractor/src/lib1/lib1.h @@ -0,0 +1,6 @@ +#ifndef LIB1_H +#define LIB1_H + +void say_something(void); + +#endif diff --git a/test cases/frameworks/38 gettext extractor/src/lib1/meson.build b/test cases/frameworks/38 gettext extractor/src/lib1/meson.build new file mode 100644 index 000000000000..0728b96eeac9 --- /dev/null +++ b/test cases/frameworks/38 gettext extractor/src/lib1/meson.build @@ -0,0 +1,3 @@ +lib1 = library('mylib1', 'lib1.c') +lib1_pot = xgettext.extract('lib1', lib1) +lib1_includes = include_directories('.') \ No newline at end of file diff --git a/test cases/frameworks/38 gettext extractor/src/lib2/lib2.c b/test cases/frameworks/38 gettext extractor/src/lib2/lib2.c new file mode 100644 index 000000000000..051271ec703d --- /dev/null +++ b/test cases/frameworks/38 gettext extractor/src/lib2/lib2.c @@ -0,0 +1,13 @@ +#include "lib2.h" + +#include + +#include + +#define tr(STRING) (STRING) + +void say_something_else(void) +{ + say_something(); + printf("%s\n", tr("Something else!")); +} diff --git a/test cases/frameworks/38 gettext extractor/src/lib2/lib2.h b/test cases/frameworks/38 gettext extractor/src/lib2/lib2.h new file mode 100644 index 000000000000..faf693f7ceb3 --- /dev/null +++ b/test cases/frameworks/38 gettext extractor/src/lib2/lib2.h @@ -0,0 +1,6 @@ +#ifndef LIB2_H +#define LIB2_H + +void say_something_else(void); + +#endif diff --git a/test cases/frameworks/38 gettext extractor/src/lib2/meson.build b/test cases/frameworks/38 gettext extractor/src/lib2/meson.build new file mode 100644 index 000000000000..137f46e9dc2f --- /dev/null +++ b/test cases/frameworks/38 gettext extractor/src/lib2/meson.build @@ -0,0 +1,3 @@ +lib2 = library('mylib2', 'lib2.c', include_directories: lib1_includes, link_with: lib1) +lib2_pot = xgettext.extract('lib2', lib2) +lib2_includes = include_directories('.') diff --git a/test cases/frameworks/38 gettext extractor/src/main.c b/test cases/frameworks/38 gettext extractor/src/main.c new file mode 100644 index 000000000000..807096bd7925 --- /dev/null +++ b/test cases/frameworks/38 gettext extractor/src/main.c @@ -0,0 +1,8 @@ +#include + +int main(void) +{ + say_something_else(); + + return 0; +} diff --git a/test cases/frameworks/38 gettext extractor/src/meson.build b/test cases/frameworks/38 gettext extractor/src/meson.build new file mode 100644 index 000000000000..81c059c6bddb --- /dev/null +++ b/test cases/frameworks/38 gettext extractor/src/meson.build @@ -0,0 +1,6 @@ +subdir('lib1') +subdir('lib2') + +main = executable('say', 'main.c', link_with: [lib2], include_directories: lib2_includes) + +main_pot = xgettext.extract('main', main, install: true, install_dir: 'intl', install_tag: 'intl', merge: true) diff --git a/test cases/frameworks/38 gettext extractor/test.json b/test cases/frameworks/38 gettext extractor/test.json new file mode 100644 index 000000000000..e69373a8c237 --- /dev/null +++ b/test cases/frameworks/38 gettext extractor/test.json @@ -0,0 +1,6 @@ +{ + "installed": [ + { "type": "file", "file": "usr/intl/main.pot" } + ], + "expect_skip_on_jobname": ["azure", "cygwin"] +} \ No newline at end of file