Lacework tester failing on one environment

[mikemcd3912]

Describe the bug
Lacework pods appear to deploy correctly, however the tester isn't completing successfully on one of our EKS environment's testing - @jefferyfry Can you assist us with troubleshooting what may be going wrong?

Screenshots

pod/lacework-agent-6shq4 condition met
pod/lacework-agent-79tvr condition met
pod/lacework-agent-8f498 condition met
pod/lacework-agent-s9kkp condition met
Kubectl Logs from Agent:
 time="2024-10-23T21:00:37.722Z" level=info msg="Discovered containers: 22 (new: 0) reason ticker" caller="cmap.go:1017" pid=2563418
time="2024-10-23T21:01:37.723Z" level=info msg="Discovered containers: 22 (new: 0) reason ticker" caller="cmap.go:1017" pid=2563418
time="2024-10-23T21:02:35.588Z" level=info msg="Virtpcap-Stats: \"ebpflite-stats\": {\"NumUnknownCloses\":10764,\"NumMissingCloses\":1585,\"NumActiveConnections\":79,\"ConnMsgs\":24649,\"ConnMsgsSkip\":3035,\"CniXformIps\":49,\"ConnEvents\":16412,\"ExitEvents\":25466}" caller="virt_pcap.go:678" pid=2563418
time="2024-10-23T21:02:37.722Z" level=info msg="Discovered containers: 22 (new: 0) reason ticker" caller="cmap.go:1017" pid=2563418
time="2024-10-23T21:03:31.549Z" level=info msg="Payload Total : [2835253], Curr : [1599]" caller="rpc.go:146" pid=2563418
time="2024-10-23T21:03:34.497Z" level=info msg="Iplocal-Stats: {ContAddrs:51,HostAddrs:8,LocalAddrs:544,NetlinkEvents:28,NumInterfaces:20,RefreshAddrsAsync:19,RouteMapNodesCur:527,RouteMapNodesTotal:165478,RouteMapsBuilt:314,RouteMapsErr:314, CNI: :notk8s,Interfaces: [{Name: lxc5087a9ed1c16,Type: container-if},{Name: ens192,Type: external-if},{Name: cilium_net,Type: internal-if},{Name: cilium_host,Type: internal-if},{Name: lxcd6a077b589e1,Type: container-if},{Name: cilium_vxlan,Type: internal-if},{Name: lxca48d0ccd09c6,Type: container-if},{Name: lxcb0f879a1c4f5,Type: container-if},{Name: lxc8aa796eec71c,Type: container-if},{Name: lxcb0c17d41e0cd,Type: container-if},{Name: lxc3a0f33ce28ec,Type: container-if},{Name: lxc4fd1d21b0a9a,Type: container-if},{Name: lxc_health,Type: container-if},{Name: lxceb876f6d8825,Type: container-if},{Name: lxc2608882b9886,Type: container-if},{Name: lxc88af7dfb946b,Type: container-if},{Name: any,Type: unknown-if},{Name: lo,Type: internal-if},{Name: nflog,Type: unknown-if},{Name: nfqueue,Type: unknown-if}],Container IPs: [10.86.87.171,10.86.87.182,10.86.87.208,10.86.87.169,10.86.87.214,10.86.87.173,10.86.87.139,10.86.87.252,10.86.87.204,10.86.87.212,10.86.87.224,10.86.87.174,10.86.87.196,10.86.87.147,10.86.87.253,10.86.87.143,10.86.87.136,10.86.87.248,10.86.87.195]}" caller="iplocal.go:303" pid=2563418
time="2024-10-23T21:03:35.091Z" level=info msg="PTE Stats: {\"LostSamples\":1708,\"NumEvents\":115163,\"NumListens\":273,\"NumConnects\":15886,\"NumConnectsNoPidHash\":9,\"NumAccepts\":13864,\"NumEstablished\":28752,\"NumCloses\":29799,\"NumListenCloses\":51,\"NumListenStaleCloses\":1,\"NumValidateStaleListens\":18,\"NumLoopBackContListens\":15,\"NumPidHashCacheHits\":15747,\"NumAcceptsNoPids\":304,\"NumAcceptsListenCollision\":46,\"NumAcceptsListenNoContainerIp\":86,\"LostKernStatsSamples\":240,\"NumKernelStatsEvents\":1237,\"NumKernelErrPosts\":1237,\"LastKernelErr\":-28,\"NumShortLivedProcs\":319,\"EbpfEnabled\":true,\"EbpfPrograms\":[[\"inet_sock_set_state_v2\",\"sched_process_exit\",\"tcp4_seq_show\",\"tcp6_seq_show\"]]}, numGlListens 8, numIpListens 44, skListens 32" caller="pte.go:1837" pid=2563418
time="2024-10-23T21:03:37.723Z" level=info msg="Discovered containers: 22 (new: 0) reason ticker" caller="cmap.go:1017" pid=2563418
time="2024-10-23T21:04:35.091Z" level=info msg="Con-State Stats: active-con 896, active-tuples 888, stat-counts: {\"NumPEvents\":29704,\"NumTEvents\":29019,\"NumEEvents\":30012,\"NumTBeforePEvents\":333,\"NumEEventMisses\":3195,\"InvalidDirEvents\":16843,\"NumActiveTupleReuses\":6,\"NumClosedTupleReuses\":55,\"NumPidReuses\":294,\"NumExpiredActive\":2990,\"NumExpiredClosed\":25949,\"NumExpiredActiveNoLkup\":2990,\"NumExpiredClosedNoLkup\":25949,\"NumPidHashCacheHit\":29328,\"NumProcMapHit\":27,\"NumForceCmapDiscoveries\":10,\"NumSkForceCmapDiscoveries\":12}" caller="connstate.go:552" pid=2563418
time="2024-10-23T21:04:37.723Z" level=info msg="Discovered containers: 22 (new: 0) reason ticker" caller="cmap.go:1017" pid=2563418
time="2024-10-23T21:01:49.720Z" level=info msg="Discovered containers: 30 (new: 0) reason ticker" caller="cmap.go:1017" pid=1876985
time="2024-10-23T21:02:49.720Z" level=info msg="Discovered containers: 30 (new: 0) reason ticker" caller="cmap.go:1017" pid=1876985
time="2024-10-23T21:03:46.614Z" level=info msg="Iplocal-Stats: {ContAddrs:54,HostAddrs:8,LocalAddrs:546,NetlinkEvents:26,NumInterfaces:23,RefreshAddrsAsync:16,RouteMapNodesCur:527,RouteMapNodesTotal:78523,RouteMapsBuilt:149,RouteMapsErr:149, CNI: :notk8s,Interfaces: [{Name: ens192,Type: external-if},{Name: cilium_net,Type: internal-if},{Name: cilium_host,Type: internal-if},{Name: cilium_vxlan,Type: internal-if},{Name: lxc_health,Type: container-if},{Name: lxc614c3f345cf5,Type: container-if},{Name: lxc176cdc3989d8,Type: container-if},{Name: lxcdf562cc5bf8b,Type: container-if},{Name: lxcf11ee9d457f4,Type: container-if},{Name: lxc8a3ab47ee6c7,Type: container-if},{Name: lxcebe10a71c6a4,Type: container-if},{Name: lxcffc7ed2fddfb,Type: container-if},{Name: lxc336c197a9346,Type: container-if},{Name: lxc399b4947ebc9,Type: container-if},{Name: lxc51945c19f5d8,Type: container-if},{Name: lxce75c77eaa01e,Type: container-if},{Name: lxcdf251a4558f1,Type: container-if},{Name: lxc45346cee24ca,Type: container-if},{Name: lxc423f4c387c1b,Type: container-if},{Name: any,Type: unknown-if},{Name: lo,Type: internal-if},{Name: nflog,Type: unknown-if},{Name: nfqueue,Type: unknown-if}],Container IPs: [10.86.86.158,10.86.86.228,10.86.86.185,10.86.86.216,10.86.86.230,10.86.86.193,10.86.86.154,10.86.86.180,10.86.86.218,10.86.86.217,10.86.86.182,10.86.86.250,10.86.86.130,10.86.86.229,10.86.86.201,10.86.86.129,10.86.86.176,10.86.86.231,10.86.86.151,10.86.86.147]}" caller="iplocal.go:303" pid=1876985
time="2024-10-23T21:03:47.399Z" level=info msg="Con-State Stats: active-con 1267, active-tuples 1256, stat-counts: {\"NumPEvents\":74046,\"NumTEvents\":53151,\"NumEEvents\":74208,\"NumTBeforePEvents\":637,\"NumEEventMisses\":1162,\"NumDupTupleEvents\":1,\"NumDupPidEvents\":1,\"InvalidDirEvents\":13865,\"NumActiveTupleReuses\":13,\"NumClosedTupleReuses\":104,\"NumPidReuses\":339,\"NumExpiredActive\":994,\"NumExpiredClosed\":71811,\"NumExpiredActiveNoLkup\":994,\"NumExpiredClosedNoLkup\":71811,\"NumPidHashCacheHit\":73662,\"NumProcMapHit\":35,\"NumForceCmapDiscoveries\":2}" caller="connstate.go:552" pid=1876985
time="2024-10-23T21:03:47.400Z" level=info msg="PTE Stats: {\"LostSamples\":3738,\"NumEvents\":360007,\"NumListens\":270,\"NumConnects\":49180,\"NumConnectsNoPidHash\":23,\"NumAccepts\":25096,\"NumEstablished\":53151,\"NumCloses\":74222,\"NumListenCloses\":14,\"NumListenPortConflictCloses\":1,\"NumLoopBackContListens\":20,\"NumPidHashCacheHits\":48928,\"NumAcceptsNoPids\":207,\"NumAcceptsListenNoContainerIp\":13,\"LostKernStatsSamples\":1608,\"NumKernelStatsEvents\":2077,\"NumKernelErrPosts\":2077,\"LastKernelErr\":-28,\"NumShortLivedProcs\":385,\"EbpfEnabled\":true,\"EbpfPrograms\":[[\"inet_sock_set_state_v2\",\"sched_process_exit\",\"tcp4_seq_show\",\"tcp6_seq_show\"]]}, numGlListens 6, numIpListens 81, skListens 50" caller="pte.go:1837" pid=1876985
time="2024-10-23T21:03:48.002Z" level=info msg="Virtpcap-Stats: \"ebpflite-stats\": {\"NumUnknownCloses\":23903,\"NumMissingCloses\":583,\"NumActiveConnections\":52,\"ConnMsgs\":31597,\"ConnMsgsSkip\":4607,\"CniXformIps\":24,\"ConnEvents\":28935,\"ExitEvents\":52174}" caller="virt_pcap.go:678" pid=1876985
time="2024-10-23T21:03:49.721Z" level=info msg="Discovered containers: 30 (new: 0) reason ticker" caller="cmap.go:1017" pid=1876985
time="2024-10-23T21:04:43.794Z" level=info msg="Payload Total : [8902929], Curr : [3521]" caller="rpc.go:146" pid=1876985
time="2024-10-23T21:04:47.443Z" level=warning msg="agentctrl/v1/config/agent/current send error: err: http error: [404] Not Found" caller="lwctrltransport.go:276" pid=1876985
time="2024-10-23T21:04:49.721Z" level=info msg="Discovered containers: 30 (new: 0) reason ticker" caller="cmap.go:1017" pid=1876985
time="2024-10-23T21:03:37.392Z" level=info msg="Start capture on : lxc1c1971e43284, len 128" caller="netflow.go:562" pid=2246042
time="2024-10-23T21:03:37.405Z" level=info msg="intf: lxc1c1971e43284 filter [not tcp]" caller="capture.go:190" pid=2246042
time="2024-10-23T21:03:37.406Z" level=info msg="Processing packets on: lxc1c1971e43284, intf_type container-if, configured vxlanPorts: []" caller="capture.go:281" pid=2246042
time="2024-10-23T21:03:37.434Z" level=info msg="Discovered containers: 17 (new: 1) reason force-scan" caller="cmap.go:1017" pid=2246042
time="2024-10-23T21:04:10.450Z" level=info msg="Pkt (lxc1c1971e43284)-Stats:{CTMap:1277,DNS-MultiQs:0,Decode-DNS-Err:0,Drop-Dns-ErrResp:0,DropMal:1,Empty-DNS:0,Pcap:10,PcapSynAck:0,PcapSyn:0,PcapAck:0,PcapFin:0,PcapRst:0,PcapUnk:0,Sink-Conn-TCP:0,Sink-Conn-TCP-U:0,Sink-Conn-UDP:0,Sink-Conn-UDP-U:0,Sink-DNS:0,OtherPrtcols:9,PCAPTickerProTime:{Name:lxc1c1971e43284 PCAP ticker time in ms,UpperBounds:[1,2,4,8,16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,131072,262144,524288,1048576],Counts:[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],CumSum:0,TotalCount:1,AvgVal:0}}" caller="interface.go:1272" pid=2246042
time="2024-10-23T21:04:10.450Z" level=info msg="Stopped processing on: [lxc1c1971e43284] [The interface disappeared] [-1]" caller="capture.go:376" pid=2246042
time="2024-10-23T21:04:20.852Z" level=info msg="Start capture on : lxc33cf7f76a25c, len 128" caller="netflow.go:562" pid=2246042
time="2024-10-23T21:04:20.865Z" level=info msg="intf: lxc33cf7f76a25c filter [not tcp]" caller="capture.go:190" pid=2246042
time="2024-10-23T21:04:20.865Z" level=info msg="Processing packets on: lxc33cf7f76a25c, intf_type container-if, configured vxlanPorts: []" caller="capture.go:281" pid=2246042
time="2024-10-23T21:04:38.856Z" level=info msg="Discovered containers: 17 (new: 0) reason ticker" caller="cmap.go:1017" pid=2246042
time="2024-10-23T21:01:16.852Z" level=info msg="Payload Total : [3959585], Curr : [1822]" caller="rpc.go:146" pid=2634558
time="2024-10-23T21:01:37.749Z" level=info msg="Discovered containers: 26 (new: 0) reason ticker" caller="cmap.go:1017" pid=2634558
time="2024-10-23T21:02:37.749Z" level=info msg="Discovered containers: 26 (new: 0) reason ticker" caller="cmap.go:1017" pid=2634558
time="2024-10-23T21:03:34.803Z" level=info msg="Iplocal-Stats: {ContAddrs:59,HostAddrs:8,LocalAddrs:549,NetlinkEvents:34,NumInterfaces:24,RefreshAddrsAsync:20,RouteMapNodesCur:527,RouteMapNodesTotal:145979,RouteMapsBuilt:277,RouteMapsErr:277, CNI: :notk8s,Interfaces: [{Name: ens192,Type: external-if},{Name: cilium_net,Type: internal-if},{Name: cilium_host,Type: internal-if},{Name: cilium_vxlan,Type: internal-if},{Name: lxc83c5edc19686,Type: container-if},{Name: lxcdd713807833b,Type: container-if},{Name: lxc886703e2c0b8,Type: container-if},{Name: lxcfe33ea51bc59,Type: container-if},{Name: lxc8c60221a3ba0,Type: container-if},{Name: lxc8ca1287a3cf0,Type: container-if},{Name: lxc_health,Type: container-if},{Name: lxc6a5fbc334080,Type: container-if},{Name: lxc6415bf207f5e,Type: container-if},{Name: lxc4dd2c2798d02,Type: container-if},{Name: lxcc15b8a815f45,Type: container-if},{Name: lxceff23eaba03e,Type: container-if},{Name: lxc468c4b4d1d43,Type: container-if},{Name: lxc6aaa29453d5b,Type: container-if},{Name: lxca338e175fadd,Type: container-if},{Name: lxc6694595fa7ab,Type: container-if},{Name: any,Type: unknown-if},{Name: lo,Type: internal-if},{Name: nflog,Type: unknown-if},{Name: nfqueue,Type: unknown-if}],Container IPs: [10.86.87.16,10.86.87.24,10.86.87.81,10.86.87.92,10.86.87.65,10.86.87.28,10.86.87.124,10.86.87.4,10.86.87.120,10.86.87.5,10.86.87.74,10.86.87.39,10.86.87.45,10.86.87.110,10.86.87.123,10.86.87.15,10.86.87.121,10.86.87.47,10.86.87.40,10.86.87.115,10.86.87.2,10.86.87.119]}" caller="iplocal.go:303" pid=2634558
time="2024-10-23T21:03:35.435Z" level=info msg="Con-State Stats: active-con 1320, active-tuples 1315, stat-counts: {\"NumPEvents\":54354,\"NumTEvents\":53456,\"NumEEvents\":55281,\"NumTBeforePEvents\":96,\"NumEEventMisses\":3427,\"NumDupTupleEvents\":1,\"NumDupPidEvents\":1,\"InvalidDirEvents\":12040,\"NumActiveTupleReuses\":26,\"NumClosedTupleReuses\":247,\"NumPidReuses\":300,\"NumExpiredActive\":2634,\"NumExpiredClosed\":50465,\"NumExpiredActiveNoLkup\":2634,\"NumExpiredClosedNoLkup\":50465,\"NumPidHashCacheHit\":53986,\"NumProcMapHit\":28,\"NumForceCmapDiscoveries\":7,\"NumSkForceCmapDiscoveries\":11}" caller="connstate.go:552" pid=2634558
time="2024-10-23T21:03:35.435Z" level=info msg="PTE Stats: {\"LostSamples\":370,\"NumEvents\":186560,\"NumListens\":301,\"NumConnects\":26774,\"NumConnectsNoPidHash\":14,\"NumAccepts\":28512,\"NumEstablished\":53456,\"NumCloses\":55333,\"NumListenCloses\":51,\"NumListenStaleCloses\":1,\"NumValidateStaleListens\":24,\"NumListenPortConflictCloses\":1,\"NumLoopBackContListens\":15,\"NumPidHashCacheHits\":26569,\"NumAcceptsNoPids\":918,\"NumAcceptsListenCollision\":392,\"NumAcceptsListenNoContainerIp\":241,\"NumListenErrNoNsIds\":4,\"NumKernelStatsEvents\":370,\"NumKernelErrPosts\":370,\"LastKernelErr\":-28,\"NumShortLivedProcs\":329,\"EbpfEnabled\":true,\"EbpfPrograms\":[[\"inet_sock_set_state_v2\",\"sched_process_exit\",\"tcp4_seq_show\",\"tcp6_seq_show\"]]}, numGlListens 8, numIpListens 52, skListens 36" caller="pte.go:1837" pid=2634558
time="2024-10-23T21:03:36.043Z" level=info msg="Virtpcap-Stats: \"ebpflite-stats\": {\"NumUnknownCloses\":23094,\"NumMissingCloses\":1555,\"NumActiveConnections\":39,\"ConnMsgs\":33824,\"ConnMsgsSkip\":2827,\"CniXformIps\":47,\"ConnEvents\":29340,\"ExitEvents\":50800}" caller="virt_pcap.go:678" pid=2634558
time="2024-10-23T21:03:37.750Z" level=info msg="Discovered containers: 26 (new: 0) reason ticker" caller="cmap.go:1017" pid=2634558
time="2024-10-23T21:04:35.516Z" level=warning msg="agentctrl/v1/config/agent/current send error: err: http error: [404] Not Found" caller="lwctrltransport.go:276" pid=2634558
time="2024-10-23T21:04:37.750Z" level=info msg="Discovered containers: 26 (new: 0) reason ticker" caller="cmap.go:1017" pid=2634558

 סּ︵סּ Stats collection is not running 
```סּ︵סּ