Many tasks are hanging forever

[benoit74]

Last Friday, I started many task on the Zimfarm.

Few completed or failed, but most are still "running" but they do not show any activity in the log (despite normally having a quite verbose log level).

I connected on one of them and:

• containers for the task are still ok
• there is almost 0 activity

> top
   2328 root      20   0    6.8m   3.2m   2.7m R   0.6   0.0   0:00.08 top
      1 root      20   0    3.7m   2.7m   2.5m S   0.0   0.0   0:00.05 mwoffliner
      7 root      20   0   54.9m  14.7m   6.5m S   0.0   0.1  11:20.81 redis-server
      9 root      20   0   54.9m  14.7m   6.5m S   0.0   0.1   0:00.00 bio_close_file
     10 root      20   0   54.9m  14.7m   6.5m S   0.0   0.1   0:00.00 bio_aof_fsync
     11 root      20   0   54.9m  14.7m   6.5m S   0.0   0.1   0:00.00 bio_lazy_free
     12 root      20   0   54.9m  14.7m   6.5m S   0.0   0.1   0:00.00 jemalloc_bg_thd
     13 root      20   0   11.6g 281.5m  52.2m S   0.0   1.8   8:42.30 node
     14 root      20   0   11.6g 281.5m  52.2m S   0.0   1.8   0:00.05 node
     15 root      20   0   11.6g 281.5m  52.2m S   0.0   1.8   0:58.95 node
     16 root      20   0   11.6g 281.5m  52.2m S   0.0   1.8   0:58.60 node
     17 root      20   0   11.6g 281.5m  52.2m S   0.0   1.8   0:58.56 node
     18 root      20   0   11.6g 281.5m  52.2m S   0.0   1.8   0:59.07 node
     19 root      20   0   11.6g 281.5m  52.2m S   0.0   1.8   0:00.00 node
     20 root      20   0   11.6g 281.5m  52.2m S   0.0   1.8   0:01.54 node
     21 root      20   0   11.6g 281.5m  52.2m S   0.0   1.8   0:01.59 node
     22 root      20   0   11.6g 281.5m  52.2m S   0.0   1.8   0:01.59 node
     23 root      20   0   11.6g 281.5m  52.2m S   0.0   1.8   0:01.66 node
    223 root      20   0   11.6g 281.5m  52.2m S   0.0   1.8   0:28.16 node
    224 root      20   0   11.6g 281.5m  52.2m S   0.0   1.8   0:25.01 node
    225 root      20   0   11.6g 281.5m  52.2m S   0.0   1.8   0:29.03 node
    226 root      20   0   11.6g 281.5m  52.2m S   0.0   1.8   0:25.43 node
    227 root      20   0   11.6g 281.5m  52.2m S   0.0   1.8   0:05.95 node
   2318 root      20   0    4.0m   3.3m   2.8m S   0.0   0.0   0:00.03 bash

> netstat -vatn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State

While #2139 might explain some of it, I strongly suspect there is another issue.