Pull/download creates and downloads a full bundle making it unnecessary slow / supporting "target file bundles" for pull/download?

[fabianlinz]

Hi,

we are working on a bigger system with several independent self-contained-systems. For enable a smooth usage of the Crowdin Editor we planed to push all translations to a single project having a folder per subsystem. In general this works fine, but always building and downloading the full bundle makes the approach noticeably slow.

Ideally it would be possible to make use of the "target file bundles" feature to limit the number of files considered for bundle creation and download (either automatically or by explicitly defining a "target file bundle". The later would probably be the easies to implement as it could work as the current pull/download command, just that not the full bundle is download but a names "target file bundle". But I did not find such an option in the current CLI.

So for now it seems we can

• (a) live with the slowliness
• (b) use a project per subsystem (consequence is duplicated configuration effort e.g. for workflows and a less convenient editor experience
• (c) stop using the CLI
• (d) use branches as suggested for similar qustions (e.g. [1] and [2]). This probably works but to me sound like a workaround.
  Build specific subdirectory of project, instead of entire project #578
  Download only 1 folder of project #346
• (e) implement the idea outlined above using "target file bundles"

Are there other options or something that we missed? Does make using "target file bundles" sense for others as well?

Thanks and best regards,

Fabian

example:

subsystem1
"files": [
{
"source": "**/*-messages.properties",
"dest": "subsystem1/%original_file_name%",
"translation": "%original_path%/%file_name%_%two_letters_code%.%file_extension%"
}
]

subsystem2

"files": [
{
"source": "**/*-messages.properties",
"dest": "subsystem2/%original_file_name%",
"translation": "%original_path%/%file_name%_%two_letters_code%.%file_extension%"
}
]

[andrii-bodnar]

Hi @fabianlinz!

By default, the crowdin download command builds the entire project (or branch) in Crowdin, and there is no other way to build a part of the project except using branches or target file bundles.

Target file bundles allow you to set up separate bundles with their own file filters, and then you could build and download each of them separately.

You can create a new bundle using either the UI or the CLI. Then you can download a specific bundle by using its unique ID.

[fabianlinz]

Hi @andrii-bodnar,

Thanks for your reply!

The thing is that we would like to keep the functionality of unpacking the downloaded bundle and copying the relevant language files to the corresponding locations.

From my perspective download/pull conveniently combines two features (1) "creating and downloading the full bundle" and (2) "applying the downloaded bundle". What I am looking for is an approach to change (1) to download a "target file bundle" instead of the full bundle, but afterwards I still want to apply the bundle.

From my perspective extending the CLI would be the best approach (unless I overlooked an existing option).

Extending the CLI would be possible in different ways IMHO (and of cause there are more approaches below)

1. let download/pull support an additional parameter for defining the name of a target file bundle that should be downloaded instead of the full bundle. But once downloaded it as today applies the changes.
2. Offer the two parts of download/pull also as separate commands which would allow combining the existing command to download a target file bundle with the new command to apply a downloaded bundle

Does such or a similar extension make sense for you? What would be the best way forward to make it a reality?

Thanks and best regards,

Fabian

[andrii-bodnar]

What I am looking for is an approach to change (1) to download a "target file bundle" instead of the full bundle, but afterwards I still want to apply the bundle.

The crowdin bundle download command does just that. When creating a target file bundle, you can specify the Resulting file pattern and the downloaded translations will be placed locally according to this pattern.

As for extending the CLI commands, I don't think it's likely to be possible. We are also limited by the Build Project Translation API method which is used in the download/pull command under the hood.

[fabianlinz]

Hi @andrii-bodnar,

sorry that I did not make this more explicit above.

I found the crowdin bundel download command and it works for what I mentioned as step (1) "download target file bundle". But step (2) to "apply the downloaded bundle" (= unpack it + identify the relevant sources based on the configuration and copy the translations to the correct translation location) is something that I am missing. crowdin bundel download also unpacks the file, but does not consider the crowdin configuration which in my case is using source + dest + translation (see also example above). If dest would not be used probably just unpacking the zip would be sufficient.

So the second part of matching the downloaded files to the configuration and then copying the files accordingly is the crucial part for me.

Best regards,

Fabian

[andrii-bodnar]

Hi @fabianlinz, thanks for the details!

Let me shed more light on how the source upload and translation download work. The translation pattern has an effect in a regular pull/download command. When uploading the source files, the CLI will set this pattern to the "Resulting file after translation export" file setting in Crowdin, so that the platform will use it when creating a zip archive with the translations. As a result, the CLI already gets the correct structure of the translation files and basically just extracts them.

For bundles, the "Resulting file after translation export" doesn't have any effect because the exported archive structure is configured during bundle creation (Resulting file pattern). In this case, the CLI will simply use the received translation file structure that comes from the Crowdin API, and the desired file structure should be configured at the bundle level.

For example, here is my test config:

"preserve_hierarchy": true

files: [
  {
    "source": "/strings.xml",
    "dest": "subsystem1/%original_file_name%",
    "translation": "/%original_path%/%file_name%_%two_letters_code%.%file_extension%"
  },
]

crowdin push:

✔️  Fetching project info    
✔️  Directory 'subsystem1'
✔️  File 'subsystem1/strings.xml'

Bundle configuration:

crowdin bundle download 118:

✔️  Building bundle (100%)    
✔️  #118 'Subsystem 1' has been successfully downloaded
✔️  Extracting archive    
✔️  translations/strings_zh.xml
✔️  translations/strings_fr.xml
✔️  translations/strings_de.xml
✔️  translations/strings_ja.xml
✔️  translations/strings_ko.xml
✔️  translations/strings_pt.xml
✔️  translations/strings_es.xml

[fabianlinz]

Hi @andrii-bodnar

thanks for the explanation. To be honest I am a little bit confused regarding how the "Resulting file after translation export" on the Crowdin server is used. What you write sounds reasonable, but so far I observed a different behaivour:

When I execute crowdin pull --keep-archive the resulting archive has the same internal structure as a full bundle downloaded via the UI. Both zip-files follow the file structure that I can also see in the Crowdin UI (the dest structure):

 subsystem1/messages_de.properties
 subsystem1/messages_fr.properties
 subsystem1/messages_pt.properties
 subsystem2/messages_de.properties
 subsystem2/messages_fr.properties
 subsystem2/messages_pt.properties
 subsystem3/messages_de.properties
 subsystem3/messages_fr.properties
 subsystem3/messages_pt.properties

The command then copies the files to the folder where the english original is also located (the configured translation location):

adapter/site-web-component/src/i18n/messages_de.properties
adapter/site-web-component/src/i18n/messages_fr.properties
adapter/site-web-component/src/i18n/messages_pt.properties

That's why I came to the conclusion that the logic to copy the translated files to the correct translation folder is part of the CLI client code. Also the source code looked more complicated than just unpacking a zip file already containing the target structure [1].

Hope you can help me understanding the missing link.

Thanks and best regards,

Fabian

[1] https://github.com/crowdin/crowdin-cli/blob/main/src/main/java/com/crowdin/cli/commands/actions/DownloadAction.java

[andrii-bodnar]

Hi Fabian,

I'm afraid there's been some misunderstanding, and we may be talking about slightly different things. When I use the "bundle" terminology, I mean the Target Files Bundles feature, which is a separate way of downloading translations from the default.

the resulting archive has the same internal structure as a full bundle downloaded via the UI

This archive will be the same whether you download the translations via the pull/download command or via the UI (Download as Zip section, not Target File Bundles).

To achieve the same behavior with the Target File Bundles, you'll need to configure the Resulting file pattern field in the bundle settings (see my previous screenshots). Could you please show your bundle configuration?

[fabianlinz]

Hi @andrii-bodnar ,

below you find a screenshot of the target file bundle configuration.

This resulting zip for the target file bundle is

 subsystem1/messages_de.properties
 subsystem1/messages_fr.properties
 subsystem1/messages_pt.properties

The zip for the full bundle is

 subsystem1/messages_de.properties
 subsystem1/messages_fr.properties
 subsystem1/messages_pt.properties
 subsystem2/messages_de.properties
 subsystem2/messages_fr.properties
 subsystem2/messages_pt.properties
 subsystem3/messages_de.properties
 subsystem3/messages_fr.properties
 subsystem3/messages_pt.properties

So from my perspective they have a compatible internal structure. That's why I would love to be able to use the smaller zip of the "target file bundle" for the second part of the crowdin download command, which somehow looks at the zip, matches it with the configuration (or somehow uses as you mentioned server side information not included it n the zip) and then copies the translated files in the zip to the corresponding location. Just extracting the zip as it's currently done by crowdin bundle download is not sufficient.

So I want the files to be copied to

adapter/site-web-component/src/i18n/messages_de.properties
adapter/site-web-component/src/i18n/messages_fr.properties
adapter/site-web-component/src/i18n/messages_pt.properties

based on the configuration for subsystem1 (as it's done for crowdin download but just for the smaller target file bundle)

"files": [
{
"source": "**/*-messages.properties",
"dest": "subsystem1/%original_file_name%",
"translation": "%original_path%/%file_name%_%two_letters_code%.%file_extension%"
}
]

Best regards,

Fabian

[andrii-bodnar]

Thanks a lot for the additional details! Could you please also share the full local path of your source files? Is it the same as the translation path? (adapter/site-web-component/src/i18n/messages_en.properties)

[andrii-bodnar]

I just did a quick test with the following source file structure:

crowdin.yml:

"preserve_hierarchy": true

files: [
  {
    "source": "**/messages.properties",
    "dest": "subsystem1/%original_file_name%",
    "translation": "%original_path%/%file_name%_%two_letters_code%.%file_extension%"
  },
]

The default download works fine. In the case of target file bundles, the "Resulting file pattern" should be more explicit without the %original_path% placeholder. In my test, the adapter/site-web-component/src/i18n/messages_%two_letters_code%.properties works as expected and downloads the target file bundle translations to the correct location:

✔️  Building bundle (100%)    
✔️  #120 'Subsystem 1 (updated)' has been successfully downloaded
✔️  Extracting archive    
✔️  adapter/site-web-component/src/i18n/messages_zh.properties
✔️  adapter/site-web-component/src/i18n/messages_fr.properties
✔️  adapter/site-web-component/src/i18n/messages_de.properties
✔️  adapter/site-web-component/src/i18n/messages_ja.properties
✔️  adapter/site-web-component/src/i18n/messages_ko.properties
✔️  adapter/site-web-component/src/i18n/messages_pt.properties
✔️  adapter/site-web-component/src/i18n/messages_es.properties

The reason for this is indeed some underlying logic of the pull and bundle download commands. The second one is simpler and doesn't have the logic of the %original_path% placeholder handling.

I'm afraid that introducing such a change might be backward incompatible with the existing integrations.

[fabianlinz]

Hi @andrii-bodnar ,

Thank you for the time you took responding in this thread.

Thanks a lot for the additional details! Could you please also share the full local path of your source files? Is it the same as the translation path? (adapter/site-web-component/src/i18n/messages_en.properties)
yes. Source and translation are in the same location in our case. Ideally we keep the structure on Crowdin a little bit simpler without the technical details, that's why we wanted to use the dest configuration.

I agree that changing the default behaviour of 'crowdin bundle download' would be a breaking change and probably in many cases the current behaviour is exactly what's needed. Adding an addition parameter to opt into an extended unpacking approach might be an option. I think the other options mentioned options could allow the CLI to support this use case. Having a dedicated command for "unpacking a zip based on the configuration" (basically step (2) of the crowdin download) independent of were the zip is coming from (as long as it has the right internal structure) would currently be my favourite. Ideally the Crowdin server would allow to build part's of a projects based on some pattern that can be provided to crowdin download, but that's quite a change.

Reconsidering the options currently available I think we will use the branch feature for now. The wording is a little bit irritating, but beside that it does what we want (crowdin download only downloads the files relevant for a certain context) and has no practical downsides.