Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to ICU tokenizer #939

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Use ICU system package

d585a63
Select commit
Loading
Failed to load commit list.
Open

Switch to ICU tokenizer #939

Use ICU system package
d585a63
Select commit
Loading
Failed to load commit list.
firefoxci-taskcluster / clean-mono-news-crawl-en-news_2007-mono-trg succeeded Nov 22, 2024 in 5m 40s

FirefoxCI (pull_request)

Clean news-crawl news_2007 dataset mono-trg en

Details

View task in Taskcluster | View logs in Taskcluster | View task group in Taskcluster

Task Status

Started: 2024-11-22T22:55:15.313Z
Resolved: 2024-11-22T22:56:10.877Z
Task Execution Time: 55 seconds, 564 milliseconds
Task Status: completed
Reason Resolved: completed
RunId: 0

Artifacts

- public/build/news_2007.en.clean.debug.txt
- public/build/news_2007.en.zst
- public/logs/live_backing.log
- public/logs/live.log


[taskcluster 2024-11-22 22:55:15.364Z] Task ID: YgHPpWXiQ1GZN_fLFdNjQw
[taskcluster 2024-11-22 22:55:15.364Z] Worker ID: 4140703635898938734
[taskcluster 2024-11-22 22:55:15.364Z] Worker Group: us-central1-b
[taskcluster 2024-11-22 22:55:15.364Z] Worker Node Type: projects/887720501152/machineTypes/n2-highmem-32
[taskcluster 2024-11-22 22:55:15.364Z] Worker Pool: translations-1/b-linux-large-gcp-300gb
[taskcluster 2024-11-22 22:55:15.364Z] Worker Version: 38.0.5
[taskcluster 2024-11-22 22:55:15.364Z] Public IP: 34.171.74.88
[taskcluster 2024-11-22 22:55:15.365Z] Hostname: translations-1-b-linux-large-gcp-300gb-c09shsliskqkwnon7l83rq
[taskcluster 2024-11-22 22:55:15.365Z] using cache "translations-level-1-checkouts-v3-7afeb851dd97df8f3607-KnyIE1GvSz67R9mjL97Now" -> /builds/worker/checkouts

[taskcluster 2024-11-22 22:55:21.061Z] Downloading artifact "public/image.tar.zst" from task ID: KnyIE1GvSz67R9mjL97Now.
[taskcluster 2024-11-22 22:55:24.362Z] Downloaded artifact successfully.
[taskcluster 2024-11-22 22:55:24.363Z] Downloaded 287.207 mb
[taskcluster 2024-11-22 22:55:24.364Z] Decompressing downloaded image
[taskcluster 2024-11-22 22:55:26.598Z] Loading docker image from downloaded archive.
[taskcluster 2024-11-22 22:55:54.005Z] Image 'public/image.tar.zst' from task 'KnyIE1GvSz67R9mjL97Now' loaded.  Using image ID sha256:d31e1900b8212f46ff27eab4217df610f5d7a124bb4975b4b8ea07a64443f3ba.
[taskcluster 2024-11-22 22:55:54.158Z] === Task Starting ===
[setup 2024-11-22T22:55:54.484Z] run-task started in /builds/worker
[setup 2024-11-22T22:55:54.484Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c $VCS_PATH/pipeline/clean/clean-mono.sh en $MOZ_FETCHES_DIR/news_2007 $TASK_WORKDIR/artifacts/news_2007 auto news-crawl_news.2007
[setup 2024-11-22T22:55:54.484Z] Python version: 3.10.12

...(2497 lines hidden)...

[task 2024-11-22T22:56:06.149Z] 119750K .......... .......... .......... .......... .......... 93%  239M 0s
[task 2024-11-22T22:56:06.149Z] 119800K .......... .......... .......... .......... .......... 93%  148M 0s
[task 2024-11-22T22:56:06.149Z] 119850K .......... .......... .......... .......... .......... 93%  167M 0s
[task 2024-11-22T22:56:06.150Z] 119900K .......... .......... .......... .......... .......... 93%  115M 0s
[task 2024-11-22T22:56:06.150Z] 119950K .......... .......... .......... .......... .......... 93%  222M 0s
[task 2024-11-22T22:56:06.150Z] 120000K .......... .......... .......... .......... .......... 93%  264M 0s
[task 2024-11-22T22:56:06.150Z] 120050K .......... .......... .......... .......... .......... 93%  144M 0s
[task 2024-11-22T22:56:06.151Z] 120100K .......... .......... .......... .......... .......... 93%  147M 0s
[task 2024-11-22T22:56:06.151Z] 120150K .......... .......... .......... .......... .......... 93%  168M 0s
[task 2024-11-22T22:56:06.151Z] 120200K .......... .......... .......... .......... .......... 93%  235M 0s
[task 2024-11-22T22:56:06.151Z] 120250K .......... .......... .......... .......... .......... 93%  203M 0s
[task 2024-11-22T22:56:06.152Z] 120300K .......... .......... .......... .......... .......... 93%  215M 0s
[task 2024-11-22T22:56:06.152Z] 120350K .......... .......... .......... .......... .......... 93%  170M 0s
[task 2024-11-22T22:56:06.152Z] 120400K .......... .......... .......... .......... .......... 93%  144M 0s
[task 2024-11-22T22:56:06.152Z] 120450K .......... .......... .......... .......... .......... 94%  223M 0s
[task 2024-11-22T22:56:06.153Z] 120500K .......... .......... .......... .......... .......... 94%  181M 0s
[task 2024-11-22T22:56:06.153Z] 120550K .......... .......... .......... .......... .......... 94%  148M 0s
[task 2024-11-22T22:56:06.153Z] 120600K .......... .......... .......... .......... .......... 94%  241M 0s
[task 2024-11-22T22:56:06.154Z] 120650K .......... .......... .......... .......... .......... 94%  129M 0s
[task 2024-11-22T22:56:06.154Z] 120700K .......... .......... .......... .......... .......... 94%  139M 0s
[task 2024-11-22T22:56:06.154Z] 120750K .......... .......... .......... .......... .......... 94%  251M 0s
[task 2024-11-22T22:56:06.155Z] 120800K .......... .......... .......... .......... .......... 94% 46.3M 0s
[task 2024-11-22T22:56:06.156Z] 120850K .......... .......... .......... .......... .......... 94%  120M 0s
[task 2024-11-22T22:56:06.156Z] 120900K .......... .......... .......... .......... .......... 94%  140M 0s
[task 2024-11-22T22:56:06.157Z] 120950K .......... .......... .......... .......... .......... 94% 84.8M 0s
[task 2024-11-22T22:56:06.157Z] 121000K .......... .......... .......... .......... .......... 94%  268M 0s
[task 2024-11-22T22:56:06.157Z] 121050K .......... .......... .......... .......... .......... 94% 90.5M 0s
[task 2024-11-22T22:56:06.158Z] 121100K .......... .......... .......... .......... .......... 94% 67.7M 0s
[task 2024-11-22T22:56:06.159Z] 121150K .......... .......... .......... .......... .......... 94% 38.7M 0s
[task 2024-11-22T22:56:06.160Z] 121200K .......... .......... .......... .......... .......... 94%  196M 0s
[task 2024-11-22T22:56:06.160Z] 121250K .......... .......... .......... .......... .......... 94%  224M 0s
[task 2024-11-22T22:56:06.160Z] 121300K .......... .......... .......... .......... .......... 94%  166M 0s
[task 2024-11-22T22:56:06.161Z] 121350K .......... .......... .......... .......... .......... 94% 53.0M 0s
[task 2024-11-22T22:56:06.161Z] 121400K .......... .......... .......... .......... .......... 94%  242M 0s
[task 2024-11-22T22:56:06.162Z] 121450K .......... .......... .......... .......... .......... 94%  132M 0s
[task 2024-11-22T22:56:06.162Z] 121500K .......... .......... .......... .......... .......... 94%  230M 0s
[task 2024-11-22T22:56:06.162Z] 121550K .......... .......... .......... .......... .......... 94% 96.4M 0s
[task 2024-11-22T22:56:06.162Z] 121600K .......... .......... .......... .......... .......... 94%  232M 0s
[task 2024-11-22T22:56:06.163Z] 121650K .......... .......... .......... .......... .......... 94%  165M 0s
[task 2024-11-22T22:56:06.163Z] 121700K .......... .......... .......... .......... .......... 94%  143M 0s
[task 2024-11-22T22:56:06.163Z] 121750K .......... .......... .......... .......... .......... 95%  191M 0s
[task 2024-11-22T22:56:06.164Z] 121800K .......... .......... .......... .......... .......... 95% 65.7M 0s
[task 2024-11-22T22:56:06.164Z] 121850K .......... .......... .......... .......... .......... 95%  214M 0s
[task 2024-11-22T22:56:06.165Z] 121900K .......... .......... .......... .......... .......... 95%  206M 0s
[task 2024-11-22T22:56:06.165Z] 121950K .......... .......... .......... .......... .......... 95%  181M 0s
[task 2024-11-22T22:56:06.165Z] 122000K .......... .......... .......... .......... .......... 95%  231M 0s
[task 2024-11-22T22:56:06.166Z] 122050K .......... .......... .......... .......... .......... 95% 56.7M 0s
[task 2024-11-22T22:56:06.166Z] 122100K .......... .......... .......... .......... .......... 95%  250M 0s
[task 2024-11-22T22:56:06.166Z] 122150K .......... .......... .......... .......... .......... 95%  242M 0s
[task 2024-11-22T22:56:06.167Z] 122200K .......... .......... .......... .......... .......... 95%  256M 0s
[task 2024-11-22T22:56:06.167Z] 122250K .......... .......... .......... .......... .......... 95%  119M 0s
[task 2024-11-22T22:56:06.167Z] 122300K .......... .......... .......... .......... .......... 95%  105M 0s
[task 2024-11-22T22:56:06.168Z] 122350K .......... .......... .......... .......... .......... 95%  240M 0s
[task 2024-11-22T22:56:06.168Z] 122400K .......... .......... .......... .......... .......... 95%  264M 0s
[task 2024-11-22T22:56:06.168Z] 122450K .......... .......... .......... .......... .......... 95%  191M 0s
[task 2024-11-22T22:56:06.169Z] 122500K .......... .......... .......... .......... .......... 95% 72.9M 0s
[task 2024-11-22T22:56:06.169Z] 122550K .......... .......... .......... .......... .......... 95%  213M 0s
[task 2024-11-22T22:56:06.169Z] 122600K .......... .......... .......... .......... .......... 95%  256M 0s
[task 2024-11-22T22:56:06.169Z] 122650K .......... .......... .......... .......... .......... 95%  178M 0s
[task 2024-11-22T22:56:06.170Z] 122700K .......... .......... .......... .......... .......... 95%  131M 0s
[task 2024-11-22T22:56:06.170Z] 122750K .......... .......... .......... .......... .......... 95%  239M 0s
[task 2024-11-22T22:56:06.170Z] 122800K .......... .......... .......... .......... .......... 95%  198M 0s
[task 2024-11-22T22:56:06.170Z] 122850K .......... .......... .......... .......... .......... 95%  217M 0s
[task 2024-11-22T22:56:06.171Z] 122900K .......... .......... .......... .......... .......... 95%  106M 0s
[task 2024-11-22T22:56:06.171Z] 122950K .......... .......... .......... .......... .......... 95%  228M 0s
[task 2024-11-22T22:56:06.171Z] 123000K .......... .......... .......... .......... .......... 95%  238M 0s
[task 2024-11-22T22:56:06.172Z] 123050K .......... .......... .......... .......... .......... 96%  142M 0s
[task 2024-11-22T22:56:06.172Z] 123100K .......... .......... .......... .......... .......... 96%  233M 0s
[task 2024-11-22T22:56:06.172Z] 123150K .......... .......... .......... .......... .......... 96%  251M 0s
[task 2024-11-22T22:56:06.172Z] 123200K .......... .......... .......... .......... .......... 96%  138M 0s
[task 2024-11-22T22:56:06.173Z] 123250K .......... .......... .......... .......... .......... 96%  117M 0s
[task 2024-11-22T22:56:06.173Z] 123300K .......... .......... .......... .......... .......... 96%  157M 0s
[task 2024-11-22T22:56:06.173Z] 123350K .......... .......... .......... .......... .......... 96%  247M 0s
[task 2024-11-22T22:56:06.174Z] 123400K .......... .......... .......... .......... .......... 96%  168M 0s
[task 2024-11-22T22:56:06.174Z] 123450K .......... .......... .......... .......... .......... 96%  163M 0s
[task 2024-11-22T22:56:06.174Z] 123500K .......... .......... .......... .......... .......... 96%  176M 0s
[task 2024-11-22T22:56:06.175Z] 123550K .......... .......... .......... .......... .......... 96%  136M 0s
[task 2024-11-22T22:56:06.175Z] 123600K .......... .......... .......... .......... .......... 96%  244M 0s
[task 2024-11-22T22:56:06.175Z] 123650K .......... .......... .......... .......... .......... 96%  185M 0s
[task 2024-11-22T22:56:06.175Z] 123700K .......... .......... .......... .......... .......... 96%  166M 0s
[task 2024-11-22T22:56:06.176Z] 123750K .......... .......... .......... .......... .......... 96% 59.9M 0s
[task 2024-11-22T22:56:06.176Z] 123800K .......... .......... .......... .......... .......... 96%  144M 0s
[task 2024-11-22T22:56:06.177Z] 123850K .......... .......... .......... .......... .......... 96%  124M 0s
[task 2024-11-22T22:56:06.177Z] 123900K .......... .......... .......... .......... .......... 96%  264M 0s
[task 2024-11-22T22:56:06.177Z] 123950K .......... .......... .......... .......... .......... 96%  143M 0s
[task 2024-11-22T22:56:06.178Z] 124000K .......... .......... .......... .......... .......... 96% 52.2M 0s
[task 2024-11-22T22:56:06.179Z] 124050K .......... .......... .......... .......... .......... 96% 61.1M 0s
[task 2024-11-22T22:56:06.179Z] 124100K .......... .......... .......... .......... .......... 96%  259M 0s
[task 2024-11-22T22:56:06.181Z] 124150K .......... .......... .......... .......... .......... 96% 38.7M 0s
[task 2024-11-22T22:56:06.181Z] 124200K .......... .......... .......... .......... .......... 96%  230M 0s
[task 2024-11-22T22:56:06.181Z] 124250K .......... .......... .......... .......... .......... 96%  121M 0s
[task 2024-11-22T22:56:06.182Z] 124300K .......... .......... .......... .......... .......... 97% 72.5M 0s
[task 2024-11-22T22:56:06.182Z] 124350K .......... .......... .......... .......... .......... 97%  235M 0s
[task 2024-11-22T22:56:06.182Z] 124400K .......... .......... .......... .......... .......... 97%  163M 0s
[task 2024-11-22T22:56:06.183Z] 124450K .......... .......... .......... .......... .......... 97%  105M 0s
[task 2024-11-22T22:56:06.183Z] 124500K .......... .......... .......... .......... .......... 97%  133M 0s
[task 2024-11-22T22:56:06.183Z] 124550K .......... .......... .......... .......... .......... 97%  183M 0s
[task 2024-11-22T22:56:06.184Z] 124600K .......... .......... .......... .......... .......... 97%  172M 0s
[task 2024-11-22T22:56:06.184Z] 124650K .......... .......... .......... .......... .......... 97%  187M 0s
[task 2024-11-22T22:56:06.184Z] 124700K .......... .......... .......... .......... .......... 97%  251M 0s
[task 2024-11-22T22:56:06.185Z] 124750K .......... .......... .......... .......... .......... 97%  169M 0s
[task 2024-11-22T22:56:06.185Z] 124800K .......... .......... .......... .......... .......... 97% 51.5M 0s
[task 2024-11-22T22:56:06.186Z] 124850K .......... .......... .......... .......... .......... 97%  222M 0s
[task 2024-11-22T22:56:06.186Z] 124900K .......... .......... .......... .......... .......... 97%  222M 0s
[task 2024-11-22T22:56:06.186Z] 124950K .......... .......... .......... .......... .......... 97%  238M 0s
[task 2024-11-22T22:56:06.186Z] 125000K .......... .......... .......... .......... .......... 97%  235M 0s
[task 2024-11-22T22:56:06.187Z] 125050K .......... .......... .......... .......... .......... 97% 52.9M 0s
[task 2024-11-22T22:56:06.187Z] 125100K .......... .......... .......... .......... .......... 97%  242M 0s
[task 2024-11-22T22:56:06.188Z] 125150K .......... .......... .......... .......... .......... 97%  106M 0s
[task 2024-11-22T22:56:06.188Z] 125200K .......... .......... .......... .......... .......... 97%  237M 0s
[task 2024-11-22T22:56:06.188Z] 125250K .......... .......... .......... .......... .......... 97%  217M 0s
[task 2024-11-22T22:56:06.189Z] 125300K .......... .......... .......... .......... .......... 97%  251M 0s
[task 2024-11-22T22:56:06.189Z] 125350K .......... .......... .......... .......... .......... 97%  149M 0s
[task 2024-11-22T22:56:06.189Z] 125400K .......... .......... .......... .......... .......... 97%  122M 0s
[task 2024-11-22T22:56:06.190Z] 125450K .......... .......... .......... .......... .......... 97%  132M 0s
[task 2024-11-22T22:56:06.190Z] 125500K .......... .......... .......... .......... .......... 97%  138M 0s
[task 2024-11-22T22:56:06.190Z] 125550K .......... .......... .......... .......... .......... 97%  243M 0s
[task 2024-11-22T22:56:06.190Z] 125600K .......... .......... .......... .......... .......... 98%  228M 0s
[task 2024-11-22T22:56:06.191Z] 125650K .......... .......... .......... .......... .......... 98%  165M 0s
[task 2024-11-22T22:56:06.191Z] 125700K .......... .......... .......... .......... .......... 98%  151M 0s
[task 2024-11-22T22:56:06.191Z] 125750K .......... .......... .......... .......... .......... 98%  125M 0s
[task 2024-11-22T22:56:06.192Z] 125800K .......... .......... .......... .......... .......... 98%  237M 0s
[task 2024-11-22T22:56:06.192Z] 125850K .......... .......... .......... .......... .......... 98%  208M 0s
[task 2024-11-22T22:56:06.192Z] 125900K .......... .......... .......... .......... .......... 98%  215M 0s
[task 2024-11-22T22:56:06.193Z] 125950K .......... .......... .......... .......... .......... 98% 91.1M 0s
[task 2024-11-22T22:56:06.193Z] 126000K .......... .......... .......... .......... .......... 98%  242M 0s
[task 2024-11-22T22:56:06.193Z] 126050K .......... .......... .......... .......... .......... 98%  221M 0s
[task 2024-11-22T22:56:06.193Z] 126100K .......... .......... .......... .......... .......... 98%  204M 0s
[task 2024-11-22T22:56:06.194Z] 126150K .......... .......... .......... .......... .......... 98%  182M 0s
[task 2024-11-22T22:56:06.194Z] 126200K .......... .......... .......... .......... .......... 98%  184M 0s
[task 2024-11-22T22:56:06.194Z] 126250K .......... .......... .......... .......... .......... 98%  139M 0s
[task 2024-11-22T22:56:06.194Z] 126300K .......... .......... .......... .......... .......... 98%  148M 0s
[task 2024-11-22T22:56:06.195Z] 126350K .......... .......... .......... .......... .......... 98%  239M 0s
[task 2024-11-22T22:56:06.195Z] 126400K .......... .......... .......... .......... .......... 98%  199M 0s
[task 2024-11-22T22:56:06.195Z] 126450K .......... .......... .......... .......... .......... 98%  151M 0s
[task 2024-11-22T22:56:06.196Z] 126500K .......... .......... .......... .......... .......... 98%  127M 0s
[task 2024-11-22T22:56:06.196Z] 126550K .......... .......... .......... .......... .......... 98%  257M 0s
[task 2024-11-22T22:56:06.196Z] 126600K .......... .......... .......... .......... .......... 98%  223M 0s
[task 2024-11-22T22:56:06.196Z] 126650K .......... .......... .......... .......... .......... 98%  156M 0s
[task 2024-11-22T22:56:06.197Z] 126700K .......... .......... .......... .......... .......... 98%  176M 0s
[task 2024-11-22T22:56:06.197Z] 126750K .......... .......... .......... .......... .......... 98% 69.7M 0s
[task 2024-11-22T22:56:06.198Z] 126800K .......... .......... .......... .......... .......... 98%  240M 0s
[task 2024-11-22T22:56:06.198Z] 126850K .......... .......... .......... .......... .......... 98%  174M 0s
[task 2024-11-22T22:56:06.198Z] 126900K .......... .......... .......... .......... .......... 99%  128M 0s
[task 2024-11-22T22:56:06.199Z] 126950K .......... .......... .......... .......... .......... 99%  135M 0s
[task 2024-11-22T22:56:06.199Z] 127000K .......... .......... .......... .......... .......... 99% 53.2M 0s
[task 2024-11-22T22:56:06.200Z] 127050K .......... .......... .......... .......... .......... 99% 64.5M 0s
[task 2024-11-22T22:56:06.200Z] 127100K .......... .......... .......... .......... .......... 99%  225M 0s
[task 2024-11-22T22:56:06.202Z] 127150K .......... .......... .......... .......... .......... 99% 39.9M 0s
[task 2024-11-22T22:56:06.202Z] 127200K .......... .......... .......... .......... .......... 99% 67.6M 0s
[task 2024-11-22T22:56:06.203Z] 127250K .......... .......... .......... .......... .......... 99%  224M 0s
[task 2024-11-22T22:56:06.203Z] 127300K .......... .......... .......... .......... .......... 99%  254M 0s
[task 2024-11-22T22:56:06.203Z] 127350K .......... .......... .......... .......... .......... 99%  116M 0s
[task 2024-11-22T22:56:06.204Z] 127400K .......... .......... .......... .......... .......... 99%  137M 0s
[task 2024-11-22T22:56:06.204Z] 127450K .......... .......... .......... .......... .......... 99%  114M 0s
[task 2024-11-22T22:56:06.204Z] 127500K .......... .......... .......... .......... .......... 99%  109M 0s
[task 2024-11-22T22:56:06.205Z] 127550K .......... .......... .......... .......... .......... 99%  242M 0s
[task 2024-11-22T22:56:06.205Z] 127600K .......... .......... .......... .......... .......... 99%  209M 0s
[task 2024-11-22T22:56:06.205Z] 127650K .......... .......... .......... .......... .......... 99%  140M 0s
[task 2024-11-22T22:56:06.206Z] 127700K .......... .......... .......... .......... .......... 99%  106M 0s
[task 2024-11-22T22:56:06.206Z] 127750K .......... .......... .......... .......... .......... 99%  232M 0s
[task 2024-11-22T22:56:06.206Z] 127800K .......... .......... .......... .......... .......... 99%  247M 0s
[task 2024-11-22T22:56:06.207Z] 127850K .......... .......... .......... .......... .......... 99% 66.4M 0s
[task 2024-11-22T22:56:06.207Z] 127900K .......... .......... .......... .......... .......... 99% 88.4M 0s
[task 2024-11-22T22:56:06.208Z] 127950K .......... .......... .......... .......... .......... 99%  180M 0s
[task 2024-11-22T22:56:06.208Z] 128000K .......... .......... .......... .......... .......... 99%  177M 0s
[task 2024-11-22T22:56:06.208Z] 128050K .......... .......... .......... .......... .......... 99%  217M 0s
[task 2024-11-22T22:56:06.209Z] 128100K .......... .......... .......... .......... .......... 99% 92.3M 0s
[task 2024-11-22T22:56:06.209Z] 128150K .......... .......... .......... .........            100%  140M=1.3s
[task 2024-11-22T22:56:06.209Z] 
[task 2024-11-22T22:56:06.209Z] 2024-11-22 22:56:06 (99.0 MB/s) - ‘tools/lid.176.bin’ saved [131266198/131266198]
[task 2024-11-22T22:56:06.209Z] 
[task 2024-11-22T22:56:06.211Z] + zstdmt -dc /builds/worker/artifacts/news_2007.en.monofix.zst
[task 2024-11-22T22:56:06.211Z] + grep -P '^en\t'
[task 2024-11-22T22:56:06.211Z] + cut -f2
[task 2024-11-22T22:56:06.211Z] + zstdmt
[task 2024-11-22T22:56:06.212Z] ++ echo 32/4
[task 2024-11-22T22:56:06.212Z] ++ bc
[task 2024-11-22T22:56:06.213Z] + parallel --no-notice --pipe -k -j 8 --block 50M 'python3 tools/langid_fasttext.py'
[task 2024-11-22T22:56:06.601Z] Warning : `load_model` does not return WordVectorModel or SupervisedModel any more, but a `FastText` object which is very similar.
[task 2024-11-22T22:56:06.606Z] + echo '### Rule-based filtering'
[task 2024-11-22T22:56:06.607Z] ### Rule-based filtering
[task 2024-11-22T22:56:06.607Z] + zstdmt -dc /builds/worker/artifacts/news_2007.en.langid.zst
[task 2024-11-22T22:56:06.607Z] + parallel --no-notice --pipe -k -j 32 --block 50M 'python3 tools/clean_mono.py -l en --debug'
[task 2024-11-22T22:56:06.607Z] + zstdmt
[task 2024-11-22T22:56:06.851Z] + test -s /builds/worker/artifacts/news_2007.en.zst
[task 2024-11-22T22:56:06.851Z] + echo '### Remove data from intermediate steps'
[task 2024-11-22T22:56:06.851Z] ### Remove data from intermediate steps
[task 2024-11-22T22:56:06.851Z] + rm -rf /builds/worker/artifacts/news_2007.en.nrm.zst /builds/worker/artifacts/news_2007.en.langid.zst /builds/worker/artifacts/news_2007.en.monofix.zst
[task 2024-11-22T22:56:06.852Z] + echo '### Rule-based cleaning log written to: /builds/worker/artifacts/news_2007.en.clean.debug.txt'
[task 2024-11-22T22:56:06.852Z] ### Rule-based cleaning log written to: /builds/worker/artifacts/news_2007.en.clean.debug.txt
[task 2024-11-22T22:56:06.852Z] + echo '### Clean data is written to: /builds/worker/artifacts/news_2007.en.zst'
[task 2024-11-22T22:56:06.852Z] ### Clean data is written to: /builds/worker/artifacts/news_2007.en.zst
[task 2024-11-22T22:56:06.852Z] + echo '###### Done: Cleaning monolingual data'
[task 2024-11-22T22:56:06.852Z] ###### Done: Cleaning monolingual data
[fetches 2024-11-22T22:56:06.852Z] removing /builds/worker/fetches
[fetches 2024-11-22T22:56:06.852Z] finished
[taskcluster 2024-11-22 22:56:09.968Z] === Task Finished ===
[taskcluster 2024-11-22 22:56:10.368Z] Successful task run with exit code: 0 completed in 55.005 seconds