netem loss, affects all IPs during teardown

[stromnet]

First, thanks for excellent utility!

I've noticed that when using netem loss mode, at the moment it starts to tear down the container, all traffic on the targeted container is briefly affected, not just towards the IPs I've pointed out.

Usage:

$ docker run --rm -it  -v /var/run/docker.sock:/var/run/docker.sock gaiaadm/pumba --log-level debug netem --tc-image gaiadocker/iproute2 --duration 60s --target $(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'  RXCONTAINER) loss --percent 50 TXCONTAINER

During runtime, the loss seems only to affect traffic from TXCONTAINER to RXCONTAINER, but at the moment it stops, then all traffic is briefly dropped.

My qdisc-fu is non-existent, but perhaps something with order of teardown?

Full debug output:

$ docker run --rm -it  -v /var/run/docker.sock:/var/run/docker.sock gaiaadm/pumba --log-level debug netem --tc-image gaiadocker/iproute2 --duration 10s --target $(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'  mycontainer-e01) loss --percent 80 mycontainer-h01
DEBU[0000] using names                                   names="[mycontainer-h01]"
DEBU[0000] adding network random packet loss to all matching containers
DEBU[0000] listing matching containers                   labels="[]" limit=0 names="[mycontainer-h01]" pattern= random=false
DEBU[0000] listing containers
DEBU[0000] found container                               id=be1f846ec45b6c7933d5afb3591c29639a82e3e48306f9288b171eaa0e61c254 name=/stoic_euler
DEBU[0000] found container                               id=ba7e1f5f2665eb3e5be8d721fa1d560d81a92ccee5adfa54f6a9b046127fafc6 name=/mycontainer-h02
DEBU[0000] found container                               id=bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a name=/mycontainer-h01
DEBU[0000] found container                               id=49197dbdb7bc6bdb27d826776a1d3125c9dd190852ba0059d782bf445e780c00 name=/mycontainer-e02
DEBU[0000] found container                               id=df9109cc9f81311faa9a3aab2ae7867628a324cc436e550653b9697dc842c0b5 name=/mycontainer-e01
DEBU[0000] found container                               id=58aa2936b7a67e427a339616d5b31c0ab85c649129c06975caeb6fbc1d2ec499 name=/mycontainer-rtmprelay
DEBU[0000] adding network random packet loss for container  container="{{0xc00013d600 [] 0xc0004b2140 0xc000166800} {sha256:8cc018b8dcc7e5037907884444739225ff950f8e14beece1f4c7c4234d9c4749 [myimage:latest] [] sha256:2d245301f515aa18548763d340773be354fc4441512951564fb6eb97c4938038  2020-04-23T12:31:38.868279069Z 9a8d569e5c9580b2ad1942ba24f2747be9d57da887e583407bb3e05858468e57 0xc00044e640 19.03.8  0xc00044e780 amd64 linux  2090410654 2090410654 {map[LowerDir:/var/lib/docker/overlay2/1d1715afbb5eacc42bcf0a1510e61b429c4a1296058df3b747a59a1dce967081/diff:/var/lib/docker/overlay2/f1ae061f455e72178aa9146e7de2fefff816dc369b9c67642030fc0fce9f119c/diff:/var/lib/docker/overlay2/d9f1b4fc50d703bdd5fb29ce38cdb21300a63179afa0c365f69d4a3de24c2049/diff:/var/lib/docker/overlay2/82bc6942ce034dbb44a7a288f4a509fd090c38079163f80eb11da325f9c47957/diff:/var/lib/docker/overlay2/ee24e2c3d7e616b51407416dd262b83edba5490dda4d794a553cf925f9f7f2e9/diff:/var/lib/docker/overlay2/6691b575554b69dc183cbfd446f7a82ab3ae7a91f5b78ec7879f19566dfebe3c/diff:/var/lib/docker/overlay2/c9744293353510c98a64c9ca74ace864d3afa5cb6055299805fb3b696005ddcb/diff:/var/lib/docker/overlay2/fbf5ec739c02c35341da287f1e8f037c6f2a5d0fd4c497d1d4b958a270ca4dbc/diff:/var/lib/docker/overlay2/6d23f6ee2281f2eb81db91b1f0932732a33b47e4073e63cc0b4e18a5298b4d58/diff:/var/lib/docker/overlay2/2e9df429e3396f89d6ea83a3d1856865f40b2ec227b1e82748eb201bdd1c4795/diff:/var/lib/docker/overlay2/705bf94cb9c23632a324d066ac51b20c74c5332f7379d9bf1b1c93c17cf697e2/diff:/var/lib/docker/overlay2/8f5021c2b2bdf7cb911a767b0a8329b82f513036288493927ab9251e6c8a2787/diff:/var/lib/docker/overlay2/bef182cabe929fd81df50ca975d0d8a6ebf1d0fdb40671bb359dbf1cc5e4079b/diff:/var/lib/docker/overlay2/bf3a2a1c739ea852f0cf94ceb67edb234ddbeac6d095ed6a512b760131de84ab/diff:/var/lib/docker/overlay2/1c7b745a25aa6fe915d1fb64c9956f33ca8735011218edbae58e5c751216c28a/diff:/var/lib/docker/overlay2/0a80edf9636c327484d21886627eeb16a2ef0379317e11ad862e98073ba666e7/diff MergedDir:/var/lib/docker/overlay2/a884ba5dec8779b74475b434a0beb7b1f9f0c3268e7401d237da5600c7f70bba/merged UpperDir:/var/lib/docker/overlay2/a884ba5dec8779b74475b434a0beb7b1f9f0c3268e7401d237da5600c7f70bba/diff WorkDir:/var/lib/docker/overlay2/a884ba5dec8779b74475b434a0beb7b1f9f0c3268e7401d237da5600c7f70bba/work] overlay2} {layers [sha256:77b174a6a187b610e4699546bd973a8d1e77663796e3724318a2a4b24cb07ea0 sha256:daaec8e342b578377103c307dc2ff811f85be87e3f305979ad59aae410e43e55 sha256:ee0af15922f79cb2838885510fdfd66bc96283f3450c697c9306a7562c7eb946 sha256:4b0b7b76328b2fb363af6ad0510b97aa5f9ff82729072b36798df5a89f70195e sha256:e5814429ea51ff147b792438fe672d82add76723fa7a40e1d7d69237cdc1c0a8 sha256:085e9ca884a33fab2537ae1f8bb8b0fcd594ca4d2fd6f366a386a03ba7177b47 sha256:3c9bdb07d6c6470229775810e1a848e714b2f7e0b6e4b1e8891bfee5db9747c2 sha256:989d624784306f0fda5af3a203ccc194cddd018983b61e23d72b9608775e467b sha256:04b4dab465c8969965cac87de3020919a9d7420752fbc93b97a2771ba920338c sha256:c8a7e3c6b34b04092e42facd60d6da80a38dca4dfd430abd628d9b60912fd17a sha256:a249f9c22044ee55920c12be2a47d4bd7f27a34a84e25731fbb04de155c39337 sha256:8ed209a014a4fca1ad61e840c2db3c21830c7baacbf46fa0ef5d6d64526c9c58 sha256:b4642cdcf4f80547fe88b8d875ce5a95f86515c3cc8296b66dcce2a52042a595 sha256:babd5188d11fbe64acc1d9b755102659b81c2a8943945fdac70a3f9cd9c01662 sha256:9e2cda270f2d7258aa8a149374a7af2220d362bebfc9c20b164d3a7e5c58c8b4 sha256:cb0aa49ba3070ac70d4724cceb604bebd6327d3c64ea5b9fd80685ab63c74c0b sha256:13ebad3deef1c4e92197c1f0d6db5761c34bb93261fb828fdf0a3e42e68d8e99] } {{198979715 63723241899 <nil>}}}}"
DEBU[0000] running netem command                         duration=10s id=bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a iface=eth0 ips="[172.21.0.21/32]" name=/mycontainer-h01 netem="[loss 80.00]" pull=true tc-image=gaiadocker/iproute2
INFO[0000] running netem on container                    command="[loss 80.00]" dryrun=false duration=10s id=bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a ips="[172.21.0.21/32]" name=/mycontainer-h01 pull=true tc-image=gaiadocker/iproute2
DEBU[0000] start netem for container with IP(s) filter   IPs="[172.21.0.21/32]" dryrun=false id=bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a iface=eth0 name=/mycontainer-h01 pull=true tcimage=gaiadocker/iproute2
DEBU[0000] adding netem qdisc                            netem="qdisc add dev eth0 root handle 1: prio"
DEBU[0000] executing tc command in a separate container joining target container network namespace  args="[qdisc add dev eth0 root handle 1: prio]" container=bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a pull=true tc-image=gaiadocker/iproute2
DEBU[0000] network mode                                  network="container:bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a"
DEBU[0000] pulling tc-image                              image=gaiadocker/iproute2
DEBU[0001] &{Pulling from gaiadocker/iproute2   {0 0}}
DEBU[0001] &{Digest: sha256:d0cd5819fbeaf2e7501dcb223bb5bad4fda7c1e76ff9877baba6e559c70b6d1e   {0 0}}
DEBU[0001] &{Status: Image is up to date for gaiadocker/iproute2:latest   {0 0}}
DEBU[0001] creating tc-container                         image=gaiadocker/iproute2
DEBU[0001] tc container created, starting it             id=d1a1479bf6bf5851a2681c34a3916bccad4ae5407517a573121716fbfabcbe79
DEBU[0001] adding netem qdisc                            netem="qdisc add dev eth0 parent 1:1 handle 10: sfq"
DEBU[0001] executing tc command in a separate container joining target container network namespace  args="[qdisc add dev eth0 parent 1:1 handle 10: sfq]" container=bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a pull=true tc-image=gaiadocker/iproute2
DEBU[0001] network mode                                  network="container:bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a"
DEBU[0001] pulling tc-image                              image=gaiadocker/iproute2
DEBU[0003] &{Pulling from gaiadocker/iproute2   {0 0}}
DEBU[0003] &{Digest: sha256:d0cd5819fbeaf2e7501dcb223bb5bad4fda7c1e76ff9877baba6e559c70b6d1e   {0 0}}
DEBU[0003] &{Status: Image is up to date for gaiadocker/iproute2:latest   {0 0}}
DEBU[0003] creating tc-container                         image=gaiadocker/iproute2
DEBU[0003] tc container created, starting it             id=4cadd0b8eec163d24e567092695f4aacd24f4d14d75c91ee5534df9722aaaef5
DEBU[0003] adding netem qdisc                            netem="qdisc add dev eth0 parent 1:2 handle 20: sfq"
DEBU[0003] executing tc command in a separate container joining target container network namespace  args="[qdisc add dev eth0 parent 1:2 handle 20: sfq]" container=bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a pull=true tc-image=gaiadocker/iproute2
DEBU[0003] network mode                                  network="container:bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a"
DEBU[0003] pulling tc-image                              image=gaiadocker/iproute2
DEBU[0004] &{Pulling from gaiadocker/iproute2   {0 0}}
DEBU[0004] &{Digest: sha256:d0cd5819fbeaf2e7501dcb223bb5bad4fda7c1e76ff9877baba6e559c70b6d1e   {0 0}}
DEBU[0004] &{Status: Image is up to date for gaiadocker/iproute2:latest   {0 0}}
DEBU[0004] creating tc-container                         image=gaiadocker/iproute2
DEBU[0004] tc container created, starting it             id=b2b0b0560d0ba267f2a450ad5e9e4639ea9ea051b65f24a881724cb3e86cdc07
DEBU[0004] adding netem qdisc                            netem="qdisc add dev eth0 parent 1:3 handle 30: netem loss 80.00"
DEBU[0004] executing tc command in a separate container joining target container network namespace  args="[qdisc add dev eth0 parent 1:3 handle 30: netem loss 80.00]" container=bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a pull=true tc-image=gaiadocker/iproute2
DEBU[0004] network mode                                  network="container:bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a"
DEBU[0004] pulling tc-image                              image=gaiadocker/iproute2
DEBU[0006] &{Pulling from gaiadocker/iproute2   {0 0}}
DEBU[0006] &{Digest: sha256:d0cd5819fbeaf2e7501dcb223bb5bad4fda7c1e76ff9877baba6e559c70b6d1e   {0 0}}
DEBU[0006] &{Status: Image is up to date for gaiadocker/iproute2:latest   {0 0}}
DEBU[0006] creating tc-container                         image=gaiadocker/iproute2
DEBU[0006] tc container created, starting it             id=f555358cf246fe920b8f56d2ce74db2e5dcd72956ac661cbfb2eb84a99356631
DEBU[0006] adding netem filter                           netem="filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip dst 172.21.0.21/32 flowid 1:3"
DEBU[0006] executing tc command in a separate container joining target container network namespace  args="[filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip dst 172.21.0.21/32 flowid 1:3]" container=bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a pull=true tc-image=gaiadocker/iproute2
DEBU[0006] network mode                                  network="container:bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a"
DEBU[0006] pulling tc-image                              image=gaiadocker/iproute2
DEBU[0008] &{Pulling from gaiadocker/iproute2   {0 0}}
DEBU[0008] &{Digest: sha256:d0cd5819fbeaf2e7501dcb223bb5bad4fda7c1e76ff9877baba6e559c70b6d1e   {0 0}}
DEBU[0008] &{Status: Image is up to date for gaiadocker/iproute2:latest   {0 0}}
DEBU[0008] creating tc-container                         image=gaiadocker/iproute2
DEBU[0008] tc container created, starting it             id=c78f64838c43d7e6d709cc01bedd1dc7b6cb3d32dc1d2863b5f15df32b8c54f8



DEBU[0010] stopping netem command on abort               id=bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a iface=eth0 ips="[172.21.0.21/32]" name=/mycontainer-h01 tc-image=gaiadocker/iproute2
INFO[0010] stopping netem on container                   IPs="[172.21.0.21/32]" dryrun=false id=bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a iface=eth0 name=/mycontainer-h01 pull=true tc-image=gaiadocker/iproute2
DEBU[0010] stop netem for container                      IPs="[172.21.0.21/32]" dryrun=false id=bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a iface=eth0 name=/mycontainer-h01 pull=true tcimage=gaiadocker/iproute2
DEBU[0010] deleting netem qdisc                          netem="qdisc del dev eth0 parent 1:1 handle 10:"
DEBU[0010] executing tc command in a separate container joining target container network namespace  args="[qdisc del dev eth0 parent 1:1 handle 10:]" container=bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a pull=true tc-image=gaiadocker/iproute2
DEBU[0010] network mode                                  network="container:bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a"
DEBU[0010] pulling tc-image                              image=gaiadocker/iproute2
DEBU[0011] &{Pulling from gaiadocker/iproute2   {0 0}}
DEBU[0011] &{Digest: sha256:d0cd5819fbeaf2e7501dcb223bb5bad4fda7c1e76ff9877baba6e559c70b6d1e   {0 0}}
DEBU[0011] &{Status: Image is up to date for gaiadocker/iproute2:latest   {0 0}}
DEBU[0011] creating tc-container                         image=gaiadocker/iproute2
DEBU[0011] tc container created, starting it             id=5de13d79e4e4389499ec08e68d21d7bdcc195f8d4a372c16526aa98e2a5de3b4
DEBU[0011] deleting netem qdisc                          netem="qdisc del dev eth0 parent 1:2 handle 20:"
DEBU[0011] executing tc command in a separate container joining target container network namespace  args="[qdisc del dev eth0 parent 1:2 handle 20:]" container=bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a pull=true tc-image=gaiadocker/iproute2
DEBU[0011] network mode                                  network="container:bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a"
DEBU[0011] pulling tc-image                              image=gaiadocker/iproute2
DEBU[0013] &{Pulling from gaiadocker/iproute2   {0 0}}
DEBU[0013] &{Digest: sha256:d0cd5819fbeaf2e7501dcb223bb5bad4fda7c1e76ff9877baba6e559c70b6d1e   {0 0}}
DEBU[0013] &{Status: Image is up to date for gaiadocker/iproute2:latest   {0 0}}
DEBU[0013] creating tc-container                         image=gaiadocker/iproute2
DEBU[0013] tc container created, starting it             id=2fb385dbe9c2331ac45d8f1395ca29545e5308968be0f367b2b58e08354d0bc7
DEBU[0013] deleting netem qdisc                          netem="qdisc del dev eth0 parent 1:3 handle 30:"
DEBU[0013] executing tc command in a separate container joining target container network namespace  args="[qdisc del dev eth0 parent 1:3 handle 30:]" container=bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a pull=true tc-image=gaiadocker/iproute2
DEBU[0013] network mode                                  network="container:bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a"
DEBU[0013] pulling tc-image                              image=gaiadocker/iproute2
DEBU[0014] &{Pulling from gaiadocker/iproute2   {0 0}}
DEBU[0014] &{Digest: sha256:d0cd5819fbeaf2e7501dcb223bb5bad4fda7c1e76ff9877baba6e559c70b6d1e   {0 0}}
DEBU[0014] &{Status: Image is up to date for gaiadocker/iproute2:latest   {0 0}}
DEBU[0014] creating tc-container                         image=gaiadocker/iproute2
DEBU[0014] tc container created, starting it             id=d873b9688a849657a867e2a0c604f8b585621e53eb5ad258dbc8fff060325977
DEBU[0015] deleting netem qdisc                          netem="qdisc del dev eth0 root handle 1: prio"
DEBU[0015] executing tc command in a separate container joining target container network namespace  args="[qdisc del dev eth0 root handle 1: prio]" container=bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a pull=true tc-image=gaiadocker/iproute2
DEBU[0015] network mode                                  network="container:bb07cf5fa01b2b5d21744e90fca72814e037050d0c79d1966409582e0df9c60a"
DEBU[0015] pulling tc-image                              image=gaiadocker/iproute2
DEBU[0016] &{Pulling from gaiadocker/iproute2   {0 0}}
DEBU[0016] &{Digest: sha256:d0cd5819fbeaf2e7501dcb223bb5bad4fda7c1e76ff9877baba6e559c70b6d1e   {0 0}}
DEBU[0016] &{Status: Image is up to date for gaiadocker/iproute2:latest   {0 0}}
DEBU[0016] creating tc-container                         image=gaiadocker/iproute2
DEBU[0016] tc container created, starting it             id=10faaedbfdf3fffe6ba23411b871a4362fa1166db6fc55d7679b904ada3df7c9

[alexei-led]

@stromnet I will try to reproduce this issue

[stromnet]

For now I've resorted to a simpler shell script (quite limited scope of usage, but also allows me to mix delay & loss), and there it works fine when cleaning up everything at once using tc -s qdisc del dev $DEV root, but that can of course have side effects if there are other shaping active.

As for what is being cleaned up, it's also a bit simpler, so that could affect it..:

tc qdisc add dev $DEV root handle 1: prio
tc filter add dev $DEV parent 1:0 protocol ip prio 1 u32 match ip dst $DST_IP match ip protocol 17 0xff flowid 2:1
tc filter add dev $DEV parent 1:0 protocol ip prio 1 u32 match ip dst $DST_IP match ip protocol 1 0xff flowid 2:1
tc qdisc add dev $DEV parent 1:1 handle 2: netem $NETEM