Switch to ICU tokenizer #939
Open
firefoxci-taskcluster / clean-mono-news-crawl-en-news_2007-mono-trg
succeeded
Nov 22, 2024 in 5m 40s
FirefoxCI (pull_request)
Clean news-crawl news_2007 dataset mono-trg en
Details
View task in Taskcluster | View logs in Taskcluster | View task group in Taskcluster
Task Status
Started: 2024-11-22T22:55:15.313Z
Resolved: 2024-11-22T22:56:10.877Z
Task Execution Time: 55 seconds, 564 milliseconds
Task Status: completed
Reason Resolved: completed
RunId: 0
Artifacts
- public/build/news_2007.en.clean.debug.txt
- public/build/news_2007.en.zst
- public/logs/live_backing.log
- public/logs/live.log
[taskcluster 2024-11-22 22:55:15.364Z] Task ID: YgHPpWXiQ1GZN_fLFdNjQw
[taskcluster 2024-11-22 22:55:15.364Z] Worker ID: 4140703635898938734
[taskcluster 2024-11-22 22:55:15.364Z] Worker Group: us-central1-b
[taskcluster 2024-11-22 22:55:15.364Z] Worker Node Type: projects/887720501152/machineTypes/n2-highmem-32
[taskcluster 2024-11-22 22:55:15.364Z] Worker Pool: translations-1/b-linux-large-gcp-300gb
[taskcluster 2024-11-22 22:55:15.364Z] Worker Version: 38.0.5
[taskcluster 2024-11-22 22:55:15.364Z] Public IP: 34.171.74.88
[taskcluster 2024-11-22 22:55:15.365Z] Hostname: translations-1-b-linux-large-gcp-300gb-c09shsliskqkwnon7l83rq
[taskcluster 2024-11-22 22:55:15.365Z] using cache "translations-level-1-checkouts-v3-7afeb851dd97df8f3607-KnyIE1GvSz67R9mjL97Now" -> /builds/worker/checkouts
[taskcluster 2024-11-22 22:55:21.061Z] Downloading artifact "public/image.tar.zst" from task ID: KnyIE1GvSz67R9mjL97Now.
[taskcluster 2024-11-22 22:55:24.362Z] Downloaded artifact successfully.
[taskcluster 2024-11-22 22:55:24.363Z] Downloaded 287.207 mb
[taskcluster 2024-11-22 22:55:24.364Z] Decompressing downloaded image
[taskcluster 2024-11-22 22:55:26.598Z] Loading docker image from downloaded archive.
[taskcluster 2024-11-22 22:55:54.005Z] Image 'public/image.tar.zst' from task 'KnyIE1GvSz67R9mjL97Now' loaded. Using image ID sha256:d31e1900b8212f46ff27eab4217df610f5d7a124bb4975b4b8ea07a64443f3ba.
[taskcluster 2024-11-22 22:55:54.158Z] === Task Starting ===
[setup 2024-11-22T22:55:54.484Z] run-task started in /builds/worker
[setup 2024-11-22T22:55:54.484Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c $VCS_PATH/pipeline/clean/clean-mono.sh en $MOZ_FETCHES_DIR/news_2007 $TASK_WORKDIR/artifacts/news_2007 auto news-crawl_news.2007
[setup 2024-11-22T22:55:54.484Z] Python version: 3.10.12
...(2497 lines hidden)...
[task 2024-11-22T22:56:06.149Z] 119750K .......... .......... .......... .......... .......... 93% 239M 0s
[task 2024-11-22T22:56:06.149Z] 119800K .......... .......... .......... .......... .......... 93% 148M 0s
[task 2024-11-22T22:56:06.149Z] 119850K .......... .......... .......... .......... .......... 93% 167M 0s
[task 2024-11-22T22:56:06.150Z] 119900K .......... .......... .......... .......... .......... 93% 115M 0s
[task 2024-11-22T22:56:06.150Z] 119950K .......... .......... .......... .......... .......... 93% 222M 0s
[task 2024-11-22T22:56:06.150Z] 120000K .......... .......... .......... .......... .......... 93% 264M 0s
[task 2024-11-22T22:56:06.150Z] 120050K .......... .......... .......... .......... .......... 93% 144M 0s
[task 2024-11-22T22:56:06.151Z] 120100K .......... .......... .......... .......... .......... 93% 147M 0s
[task 2024-11-22T22:56:06.151Z] 120150K .......... .......... .......... .......... .......... 93% 168M 0s
[task 2024-11-22T22:56:06.151Z] 120200K .......... .......... .......... .......... .......... 93% 235M 0s
[task 2024-11-22T22:56:06.151Z] 120250K .......... .......... .......... .......... .......... 93% 203M 0s
[task 2024-11-22T22:56:06.152Z] 120300K .......... .......... .......... .......... .......... 93% 215M 0s
[task 2024-11-22T22:56:06.152Z] 120350K .......... .......... .......... .......... .......... 93% 170M 0s
[task 2024-11-22T22:56:06.152Z] 120400K .......... .......... .......... .......... .......... 93% 144M 0s
[task 2024-11-22T22:56:06.152Z] 120450K .......... .......... .......... .......... .......... 94% 223M 0s
[task 2024-11-22T22:56:06.153Z] 120500K .......... .......... .......... .......... .......... 94% 181M 0s
[task 2024-11-22T22:56:06.153Z] 120550K .......... .......... .......... .......... .......... 94% 148M 0s
[task 2024-11-22T22:56:06.153Z] 120600K .......... .......... .......... .......... .......... 94% 241M 0s
[task 2024-11-22T22:56:06.154Z] 120650K .......... .......... .......... .......... .......... 94% 129M 0s
[task 2024-11-22T22:56:06.154Z] 120700K .......... .......... .......... .......... .......... 94% 139M 0s
[task 2024-11-22T22:56:06.154Z] 120750K .......... .......... .......... .......... .......... 94% 251M 0s
[task 2024-11-22T22:56:06.155Z] 120800K .......... .......... .......... .......... .......... 94% 46.3M 0s
[task 2024-11-22T22:56:06.156Z] 120850K .......... .......... .......... .......... .......... 94% 120M 0s
[task 2024-11-22T22:56:06.156Z] 120900K .......... .......... .......... .......... .......... 94% 140M 0s
[task 2024-11-22T22:56:06.157Z] 120950K .......... .......... .......... .......... .......... 94% 84.8M 0s
[task 2024-11-22T22:56:06.157Z] 121000K .......... .......... .......... .......... .......... 94% 268M 0s
[task 2024-11-22T22:56:06.157Z] 121050K .......... .......... .......... .......... .......... 94% 90.5M 0s
[task 2024-11-22T22:56:06.158Z] 121100K .......... .......... .......... .......... .......... 94% 67.7M 0s
[task 2024-11-22T22:56:06.159Z] 121150K .......... .......... .......... .......... .......... 94% 38.7M 0s
[task 2024-11-22T22:56:06.160Z] 121200K .......... .......... .......... .......... .......... 94% 196M 0s
[task 2024-11-22T22:56:06.160Z] 121250K .......... .......... .......... .......... .......... 94% 224M 0s
[task 2024-11-22T22:56:06.160Z] 121300K .......... .......... .......... .......... .......... 94% 166M 0s
[task 2024-11-22T22:56:06.161Z] 121350K .......... .......... .......... .......... .......... 94% 53.0M 0s
[task 2024-11-22T22:56:06.161Z] 121400K .......... .......... .......... .......... .......... 94% 242M 0s
[task 2024-11-22T22:56:06.162Z] 121450K .......... .......... .......... .......... .......... 94% 132M 0s
[task 2024-11-22T22:56:06.162Z] 121500K .......... .......... .......... .......... .......... 94% 230M 0s
[task 2024-11-22T22:56:06.162Z] 121550K .......... .......... .......... .......... .......... 94% 96.4M 0s
[task 2024-11-22T22:56:06.162Z] 121600K .......... .......... .......... .......... .......... 94% 232M 0s
[task 2024-11-22T22:56:06.163Z] 121650K .......... .......... .......... .......... .......... 94% 165M 0s
[task 2024-11-22T22:56:06.163Z] 121700K .......... .......... .......... .......... .......... 94% 143M 0s
[task 2024-11-22T22:56:06.163Z] 121750K .......... .......... .......... .......... .......... 95% 191M 0s
[task 2024-11-22T22:56:06.164Z] 121800K .......... .......... .......... .......... .......... 95% 65.7M 0s
[task 2024-11-22T22:56:06.164Z] 121850K .......... .......... .......... .......... .......... 95% 214M 0s
[task 2024-11-22T22:56:06.165Z] 121900K .......... .......... .......... .......... .......... 95% 206M 0s
[task 2024-11-22T22:56:06.165Z] 121950K .......... .......... .......... .......... .......... 95% 181M 0s
[task 2024-11-22T22:56:06.165Z] 122000K .......... .......... .......... .......... .......... 95% 231M 0s
[task 2024-11-22T22:56:06.166Z] 122050K .......... .......... .......... .......... .......... 95% 56.7M 0s
[task 2024-11-22T22:56:06.166Z] 122100K .......... .......... .......... .......... .......... 95% 250M 0s
[task 2024-11-22T22:56:06.166Z] 122150K .......... .......... .......... .......... .......... 95% 242M 0s
[task 2024-11-22T22:56:06.167Z] 122200K .......... .......... .......... .......... .......... 95% 256M 0s
[task 2024-11-22T22:56:06.167Z] 122250K .......... .......... .......... .......... .......... 95% 119M 0s
[task 2024-11-22T22:56:06.167Z] 122300K .......... .......... .......... .......... .......... 95% 105M 0s
[task 2024-11-22T22:56:06.168Z] 122350K .......... .......... .......... .......... .......... 95% 240M 0s
[task 2024-11-22T22:56:06.168Z] 122400K .......... .......... .......... .......... .......... 95% 264M 0s
[task 2024-11-22T22:56:06.168Z] 122450K .......... .......... .......... .......... .......... 95% 191M 0s
[task 2024-11-22T22:56:06.169Z] 122500K .......... .......... .......... .......... .......... 95% 72.9M 0s
[task 2024-11-22T22:56:06.169Z] 122550K .......... .......... .......... .......... .......... 95% 213M 0s
[task 2024-11-22T22:56:06.169Z] 122600K .......... .......... .......... .......... .......... 95% 256M 0s
[task 2024-11-22T22:56:06.169Z] 122650K .......... .......... .......... .......... .......... 95% 178M 0s
[task 2024-11-22T22:56:06.170Z] 122700K .......... .......... .......... .......... .......... 95% 131M 0s
[task 2024-11-22T22:56:06.170Z] 122750K .......... .......... .......... .......... .......... 95% 239M 0s
[task 2024-11-22T22:56:06.170Z] 122800K .......... .......... .......... .......... .......... 95% 198M 0s
[task 2024-11-22T22:56:06.170Z] 122850K .......... .......... .......... .......... .......... 95% 217M 0s
[task 2024-11-22T22:56:06.171Z] 122900K .......... .......... .......... .......... .......... 95% 106M 0s
[task 2024-11-22T22:56:06.171Z] 122950K .......... .......... .......... .......... .......... 95% 228M 0s
[task 2024-11-22T22:56:06.171Z] 123000K .......... .......... .......... .......... .......... 95% 238M 0s
[task 2024-11-22T22:56:06.172Z] 123050K .......... .......... .......... .......... .......... 96% 142M 0s
[task 2024-11-22T22:56:06.172Z] 123100K .......... .......... .......... .......... .......... 96% 233M 0s
[task 2024-11-22T22:56:06.172Z] 123150K .......... .......... .......... .......... .......... 96% 251M 0s
[task 2024-11-22T22:56:06.172Z] 123200K .......... .......... .......... .......... .......... 96% 138M 0s
[task 2024-11-22T22:56:06.173Z] 123250K .......... .......... .......... .......... .......... 96% 117M 0s
[task 2024-11-22T22:56:06.173Z] 123300K .......... .......... .......... .......... .......... 96% 157M 0s
[task 2024-11-22T22:56:06.173Z] 123350K .......... .......... .......... .......... .......... 96% 247M 0s
[task 2024-11-22T22:56:06.174Z] 123400K .......... .......... .......... .......... .......... 96% 168M 0s
[task 2024-11-22T22:56:06.174Z] 123450K .......... .......... .......... .......... .......... 96% 163M 0s
[task 2024-11-22T22:56:06.174Z] 123500K .......... .......... .......... .......... .......... 96% 176M 0s
[task 2024-11-22T22:56:06.175Z] 123550K .......... .......... .......... .......... .......... 96% 136M 0s
[task 2024-11-22T22:56:06.175Z] 123600K .......... .......... .......... .......... .......... 96% 244M 0s
[task 2024-11-22T22:56:06.175Z] 123650K .......... .......... .......... .......... .......... 96% 185M 0s
[task 2024-11-22T22:56:06.175Z] 123700K .......... .......... .......... .......... .......... 96% 166M 0s
[task 2024-11-22T22:56:06.176Z] 123750K .......... .......... .......... .......... .......... 96% 59.9M 0s
[task 2024-11-22T22:56:06.176Z] 123800K .......... .......... .......... .......... .......... 96% 144M 0s
[task 2024-11-22T22:56:06.177Z] 123850K .......... .......... .......... .......... .......... 96% 124M 0s
[task 2024-11-22T22:56:06.177Z] 123900K .......... .......... .......... .......... .......... 96% 264M 0s
[task 2024-11-22T22:56:06.177Z] 123950K .......... .......... .......... .......... .......... 96% 143M 0s
[task 2024-11-22T22:56:06.178Z] 124000K .......... .......... .......... .......... .......... 96% 52.2M 0s
[task 2024-11-22T22:56:06.179Z] 124050K .......... .......... .......... .......... .......... 96% 61.1M 0s
[task 2024-11-22T22:56:06.179Z] 124100K .......... .......... .......... .......... .......... 96% 259M 0s
[task 2024-11-22T22:56:06.181Z] 124150K .......... .......... .......... .......... .......... 96% 38.7M 0s
[task 2024-11-22T22:56:06.181Z] 124200K .......... .......... .......... .......... .......... 96% 230M 0s
[task 2024-11-22T22:56:06.181Z] 124250K .......... .......... .......... .......... .......... 96% 121M 0s
[task 2024-11-22T22:56:06.182Z] 124300K .......... .......... .......... .......... .......... 97% 72.5M 0s
[task 2024-11-22T22:56:06.182Z] 124350K .......... .......... .......... .......... .......... 97% 235M 0s
[task 2024-11-22T22:56:06.182Z] 124400K .......... .......... .......... .......... .......... 97% 163M 0s
[task 2024-11-22T22:56:06.183Z] 124450K .......... .......... .......... .......... .......... 97% 105M 0s
[task 2024-11-22T22:56:06.183Z] 124500K .......... .......... .......... .......... .......... 97% 133M 0s
[task 2024-11-22T22:56:06.183Z] 124550K .......... .......... .......... .......... .......... 97% 183M 0s
[task 2024-11-22T22:56:06.184Z] 124600K .......... .......... .......... .......... .......... 97% 172M 0s
[task 2024-11-22T22:56:06.184Z] 124650K .......... .......... .......... .......... .......... 97% 187M 0s
[task 2024-11-22T22:56:06.184Z] 124700K .......... .......... .......... .......... .......... 97% 251M 0s
[task 2024-11-22T22:56:06.185Z] 124750K .......... .......... .......... .......... .......... 97% 169M 0s
[task 2024-11-22T22:56:06.185Z] 124800K .......... .......... .......... .......... .......... 97% 51.5M 0s
[task 2024-11-22T22:56:06.186Z] 124850K .......... .......... .......... .......... .......... 97% 222M 0s
[task 2024-11-22T22:56:06.186Z] 124900K .......... .......... .......... .......... .......... 97% 222M 0s
[task 2024-11-22T22:56:06.186Z] 124950K .......... .......... .......... .......... .......... 97% 238M 0s
[task 2024-11-22T22:56:06.186Z] 125000K .......... .......... .......... .......... .......... 97% 235M 0s
[task 2024-11-22T22:56:06.187Z] 125050K .......... .......... .......... .......... .......... 97% 52.9M 0s
[task 2024-11-22T22:56:06.187Z] 125100K .......... .......... .......... .......... .......... 97% 242M 0s
[task 2024-11-22T22:56:06.188Z] 125150K .......... .......... .......... .......... .......... 97% 106M 0s
[task 2024-11-22T22:56:06.188Z] 125200K .......... .......... .......... .......... .......... 97% 237M 0s
[task 2024-11-22T22:56:06.188Z] 125250K .......... .......... .......... .......... .......... 97% 217M 0s
[task 2024-11-22T22:56:06.189Z] 125300K .......... .......... .......... .......... .......... 97% 251M 0s
[task 2024-11-22T22:56:06.189Z] 125350K .......... .......... .......... .......... .......... 97% 149M 0s
[task 2024-11-22T22:56:06.189Z] 125400K .......... .......... .......... .......... .......... 97% 122M 0s
[task 2024-11-22T22:56:06.190Z] 125450K .......... .......... .......... .......... .......... 97% 132M 0s
[task 2024-11-22T22:56:06.190Z] 125500K .......... .......... .......... .......... .......... 97% 138M 0s
[task 2024-11-22T22:56:06.190Z] 125550K .......... .......... .......... .......... .......... 97% 243M 0s
[task 2024-11-22T22:56:06.190Z] 125600K .......... .......... .......... .......... .......... 98% 228M 0s
[task 2024-11-22T22:56:06.191Z] 125650K .......... .......... .......... .......... .......... 98% 165M 0s
[task 2024-11-22T22:56:06.191Z] 125700K .......... .......... .......... .......... .......... 98% 151M 0s
[task 2024-11-22T22:56:06.191Z] 125750K .......... .......... .......... .......... .......... 98% 125M 0s
[task 2024-11-22T22:56:06.192Z] 125800K .......... .......... .......... .......... .......... 98% 237M 0s
[task 2024-11-22T22:56:06.192Z] 125850K .......... .......... .......... .......... .......... 98% 208M 0s
[task 2024-11-22T22:56:06.192Z] 125900K .......... .......... .......... .......... .......... 98% 215M 0s
[task 2024-11-22T22:56:06.193Z] 125950K .......... .......... .......... .......... .......... 98% 91.1M 0s
[task 2024-11-22T22:56:06.193Z] 126000K .......... .......... .......... .......... .......... 98% 242M 0s
[task 2024-11-22T22:56:06.193Z] 126050K .......... .......... .......... .......... .......... 98% 221M 0s
[task 2024-11-22T22:56:06.193Z] 126100K .......... .......... .......... .......... .......... 98% 204M 0s
[task 2024-11-22T22:56:06.194Z] 126150K .......... .......... .......... .......... .......... 98% 182M 0s
[task 2024-11-22T22:56:06.194Z] 126200K .......... .......... .......... .......... .......... 98% 184M 0s
[task 2024-11-22T22:56:06.194Z] 126250K .......... .......... .......... .......... .......... 98% 139M 0s
[task 2024-11-22T22:56:06.194Z] 126300K .......... .......... .......... .......... .......... 98% 148M 0s
[task 2024-11-22T22:56:06.195Z] 126350K .......... .......... .......... .......... .......... 98% 239M 0s
[task 2024-11-22T22:56:06.195Z] 126400K .......... .......... .......... .......... .......... 98% 199M 0s
[task 2024-11-22T22:56:06.195Z] 126450K .......... .......... .......... .......... .......... 98% 151M 0s
[task 2024-11-22T22:56:06.196Z] 126500K .......... .......... .......... .......... .......... 98% 127M 0s
[task 2024-11-22T22:56:06.196Z] 126550K .......... .......... .......... .......... .......... 98% 257M 0s
[task 2024-11-22T22:56:06.196Z] 126600K .......... .......... .......... .......... .......... 98% 223M 0s
[task 2024-11-22T22:56:06.196Z] 126650K .......... .......... .......... .......... .......... 98% 156M 0s
[task 2024-11-22T22:56:06.197Z] 126700K .......... .......... .......... .......... .......... 98% 176M 0s
[task 2024-11-22T22:56:06.197Z] 126750K .......... .......... .......... .......... .......... 98% 69.7M 0s
[task 2024-11-22T22:56:06.198Z] 126800K .......... .......... .......... .......... .......... 98% 240M 0s
[task 2024-11-22T22:56:06.198Z] 126850K .......... .......... .......... .......... .......... 98% 174M 0s
[task 2024-11-22T22:56:06.198Z] 126900K .......... .......... .......... .......... .......... 99% 128M 0s
[task 2024-11-22T22:56:06.199Z] 126950K .......... .......... .......... .......... .......... 99% 135M 0s
[task 2024-11-22T22:56:06.199Z] 127000K .......... .......... .......... .......... .......... 99% 53.2M 0s
[task 2024-11-22T22:56:06.200Z] 127050K .......... .......... .......... .......... .......... 99% 64.5M 0s
[task 2024-11-22T22:56:06.200Z] 127100K .......... .......... .......... .......... .......... 99% 225M 0s
[task 2024-11-22T22:56:06.202Z] 127150K .......... .......... .......... .......... .......... 99% 39.9M 0s
[task 2024-11-22T22:56:06.202Z] 127200K .......... .......... .......... .......... .......... 99% 67.6M 0s
[task 2024-11-22T22:56:06.203Z] 127250K .......... .......... .......... .......... .......... 99% 224M 0s
[task 2024-11-22T22:56:06.203Z] 127300K .......... .......... .......... .......... .......... 99% 254M 0s
[task 2024-11-22T22:56:06.203Z] 127350K .......... .......... .......... .......... .......... 99% 116M 0s
[task 2024-11-22T22:56:06.204Z] 127400K .......... .......... .......... .......... .......... 99% 137M 0s
[task 2024-11-22T22:56:06.204Z] 127450K .......... .......... .......... .......... .......... 99% 114M 0s
[task 2024-11-22T22:56:06.204Z] 127500K .......... .......... .......... .......... .......... 99% 109M 0s
[task 2024-11-22T22:56:06.205Z] 127550K .......... .......... .......... .......... .......... 99% 242M 0s
[task 2024-11-22T22:56:06.205Z] 127600K .......... .......... .......... .......... .......... 99% 209M 0s
[task 2024-11-22T22:56:06.205Z] 127650K .......... .......... .......... .......... .......... 99% 140M 0s
[task 2024-11-22T22:56:06.206Z] 127700K .......... .......... .......... .......... .......... 99% 106M 0s
[task 2024-11-22T22:56:06.206Z] 127750K .......... .......... .......... .......... .......... 99% 232M 0s
[task 2024-11-22T22:56:06.206Z] 127800K .......... .......... .......... .......... .......... 99% 247M 0s
[task 2024-11-22T22:56:06.207Z] 127850K .......... .......... .......... .......... .......... 99% 66.4M 0s
[task 2024-11-22T22:56:06.207Z] 127900K .......... .......... .......... .......... .......... 99% 88.4M 0s
[task 2024-11-22T22:56:06.208Z] 127950K .......... .......... .......... .......... .......... 99% 180M 0s
[task 2024-11-22T22:56:06.208Z] 128000K .......... .......... .......... .......... .......... 99% 177M 0s
[task 2024-11-22T22:56:06.208Z] 128050K .......... .......... .......... .......... .......... 99% 217M 0s
[task 2024-11-22T22:56:06.209Z] 128100K .......... .......... .......... .......... .......... 99% 92.3M 0s
[task 2024-11-22T22:56:06.209Z] 128150K .......... .......... .......... ......... 100% 140M=1.3s
[task 2024-11-22T22:56:06.209Z]
[task 2024-11-22T22:56:06.209Z] 2024-11-22 22:56:06 (99.0 MB/s) - ‘tools/lid.176.bin’ saved [131266198/131266198]
[task 2024-11-22T22:56:06.209Z]
[task 2024-11-22T22:56:06.211Z] + zstdmt -dc /builds/worker/artifacts/news_2007.en.monofix.zst
[task 2024-11-22T22:56:06.211Z] + grep -P '^en\t'
[task 2024-11-22T22:56:06.211Z] + cut -f2
[task 2024-11-22T22:56:06.211Z] + zstdmt
[task 2024-11-22T22:56:06.212Z] ++ echo 32/4
[task 2024-11-22T22:56:06.212Z] ++ bc
[task 2024-11-22T22:56:06.213Z] + parallel --no-notice --pipe -k -j 8 --block 50M 'python3 tools/langid_fasttext.py'
[task 2024-11-22T22:56:06.601Z] Warning : `load_model` does not return WordVectorModel or SupervisedModel any more, but a `FastText` object which is very similar.
[task 2024-11-22T22:56:06.606Z] + echo '### Rule-based filtering'
[task 2024-11-22T22:56:06.607Z] ### Rule-based filtering
[task 2024-11-22T22:56:06.607Z] + zstdmt -dc /builds/worker/artifacts/news_2007.en.langid.zst
[task 2024-11-22T22:56:06.607Z] + parallel --no-notice --pipe -k -j 32 --block 50M 'python3 tools/clean_mono.py -l en --debug'
[task 2024-11-22T22:56:06.607Z] + zstdmt
[task 2024-11-22T22:56:06.851Z] + test -s /builds/worker/artifacts/news_2007.en.zst
[task 2024-11-22T22:56:06.851Z] + echo '### Remove data from intermediate steps'
[task 2024-11-22T22:56:06.851Z] ### Remove data from intermediate steps
[task 2024-11-22T22:56:06.851Z] + rm -rf /builds/worker/artifacts/news_2007.en.nrm.zst /builds/worker/artifacts/news_2007.en.langid.zst /builds/worker/artifacts/news_2007.en.monofix.zst
[task 2024-11-22T22:56:06.852Z] + echo '### Rule-based cleaning log written to: /builds/worker/artifacts/news_2007.en.clean.debug.txt'
[task 2024-11-22T22:56:06.852Z] ### Rule-based cleaning log written to: /builds/worker/artifacts/news_2007.en.clean.debug.txt
[task 2024-11-22T22:56:06.852Z] + echo '### Clean data is written to: /builds/worker/artifacts/news_2007.en.zst'
[task 2024-11-22T22:56:06.852Z] ### Clean data is written to: /builds/worker/artifacts/news_2007.en.zst
[task 2024-11-22T22:56:06.852Z] + echo '###### Done: Cleaning monolingual data'
[task 2024-11-22T22:56:06.852Z] ###### Done: Cleaning monolingual data
[fetches 2024-11-22T22:56:06.852Z] removing /builds/worker/fetches
[fetches 2024-11-22T22:56:06.852Z] finished
[taskcluster 2024-11-22 22:56:09.968Z] === Task Finished ===
[taskcluster 2024-11-22 22:56:10.368Z] Successful task run with exit code: 0 completed in 55.005 seconds
Loading