Datadog not receiving metrics from container

[davidsiaw]

For some reason the containerized version of datadog is not able to get docker metrics and stats. Needs research. Logs are not receivable if logentries plugin is being used.

Errors related:

2020-04-17 07:24:40 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check cpu
2020-04-17 07:24:40 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check cpu
2020-04-17 07:24:44 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check network
2020-04-17 07:24:44 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check network
2020-04-17 07:24:45 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check load
2020-04-17 07:24:45 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check load
2020-04-17 07:24:46 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check file_handle
2020-04-17 07:24:46 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check file_handle
2020-04-17 07:24:47 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check disk
2020-04-17 07:24:47 UTC | CORE | WARN | (pkg/collector/python/datadog_agent.go:118 in LogMessage) | disk:e5dffb8bef24336f | (disk.py:77) | Unable to get disk metrics for /host/proc/sys/fs/binfmt_misc: [Errno 40] Too many levels of symbolic links: '/host/proc/sys/fs/binfmt_misc'
2020-04-17 07:24:47 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check disk
2020-04-17 07:24:49 UTC | PROCESS | WARN | (pkg/util/containers/metrics/cgroup_detect.go:126 in parseCgroupMountPoints) | No mountPoints were detected, current cgroup root is: /host/sys/fs/cgroup/
2020-04-17 07:24:49 UTC | CORE | INFO | (pkg/logs/scheduler/scheduler.go:73 in Schedule) | Received a new service: docker://c86bdf36814fd5c62036f16a03a6c1c96000bc6ff7a98e88c2f5c01b6c24dfdc
2020-04-17 07:24:49 UTC | CORE | WARN | (pkg/logs/input/docker/launcher.go:213 in startTailer) | Could not start tailer c86bdf36814fd5c62036f16a03a6c1c96000bc6ff7a98e88c2f5c01b6c24dfdc: Error response from daemon: configured logging driver does not support reading
2020-04-17 07:24:49 UTC | CORE | INFO | (pkg/logs/scheduler/scheduler.go:73 in Schedule) | Received a new service: docker://45747b7373e772da8a3f2f398c0f45165d1843f937464832bb22b53b0c9ccc78
2020-04-17 07:24:49 UTC | CORE | WARN | (pkg/logs/input/docker/launcher.go:213 in startTailer) | Could not start tailer 45747b7373e772da8a3f2f398c0f45165d1843f937464832bb22b53b0c9ccc78: Error response from daemon: configured logging driver does not support reading
2020-04-17 07:24:49 UTC | CORE | INFO | (pkg/logs/scheduler/scheduler.go:73 in Schedule) | Received a new service: docker://bde850ddb39ee1ac9896cf62e723585cfec6d44f45a47ebc620ca1265d573773
2020-04-17 07:24:49 UTC | CORE | WARN | (pkg/logs/input/docker/launcher.go:213 in startTailer) | Could not start tailer bde850ddb39ee1ac9896cf62e723585cfec6d44f45a47ebc620ca1265d573773: Error response from daemon: configured logging driver does not support reading
2020-04-17 07:24:50 UTC | CORE | INFO | (pkg/logs/scheduler/scheduler.go:73 in Schedule) | Received a new service: docker://ea346b054f4d2c8abbef8bb01e7288a477e8c4f2bff99320579b6745c9716f09
2020-04-17 07:24:50 UTC | CORE | WARN | (pkg/logs/input/docker/launcher.go:213 in startTailer) | Could not start tailer ea346b054f4d2c8abbef8bb01e7288a477e8c4f2bff99320579b6745c9716f09: Error response from daemon: configured logging driver does not support reading
2020-04-17 07:24:50 UTC | CORE | INFO | (pkg/logs/scheduler/scheduler.go:73 in Schedule) | Received a new service: docker://36c878491ce028213eb4fe07923718e6620d423416919156a61e9bc3b22c841d
2020-04-17 07:24:50 UTC | CORE | WARN | (pkg/logs/input/docker/launcher.go:213 in startTailer) | Could not start tailer 36c878491ce028213eb4fe07923718e6620d423416919156a61e9bc3b22c841d: Error response from daemon: configured logging driver does not support reading
2020-04-17 07:24:50 UTC | CORE | INFO | (pkg/logs/scheduler/scheduler.go:73 in Schedule) | Received a new service: docker://1f5d11d42809127e136d9a2f1e3abe8c75ab4fa926b62f5a9b3dde99a98fa1cf
2020-04-17 07:24:50 UTC | CORE | WARN | (pkg/logs/input/docker/launcher.go:213 in startTailer) | Could not start tailer 1f5d11d42809127e136d9a2f1e3abe8c75ab4fa926b62f5a9b3dde99a98fa1cf: Error response from daemon: configured logging driver does not support reading
2020-04-17 07:24:51 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check uptime
2020-04-17 07:24:51 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check uptime
2020-04-17 07:24:52 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check memory
2020-04-17 07:24:52 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check memory
2020-04-17 07:24:53 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check io
2020-04-17 07:24:53 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check io
2020-04-17 07:24:53 UTC | CORE | INFO | (pkg/logs/scheduler/scheduler.go:73 in Schedule) | Received a new service: docker://6e6416856180834a2ead4965625c78e8319bac789084bbc6215491d538330ada
2020-04-17 07:24:53 UTC | CORE | WARN | (pkg/logs/input/docker/launcher.go:213 in startTailer) | Could not start tailer 6e6416856180834a2ead4965625c78e8319bac789084bbc6215491d538330ada: Error response from daemon: configured logging driver does not support reading
2020-04-17 07:24:54 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check docker
2020-04-17 07:24:54 UTC | CORE | WARN | (pkg/util/containers/metrics/cgroup_detect.go:126 in parseCgroupMountPoints) | No mountPoints were detected, current cgroup root is: /host/sys/fs/cgroup/
2020-04-17 07:24:54 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check docker
2020-04-17 07:24:54 UTC | CORE | INFO | (pkg/logs/scheduler/scheduler.go:73 in Schedule) | Received a new service: docker://cb6abd4a8fd5ffbe5b2c2b1a99aab21e6dd52cbb3a011cb60cbe97f89fe16815
2020-04-17 07:24:54 UTC | CORE | WARN | (pkg/logs/input/docker/launcher.go:213 in startTailer) | Could not start tailer cb6abd4a8fd5ffbe5b2c2b1a99aab21e6dd52cbb3a011cb60cbe97f89fe16815: Error response from daemon: configured logging driver does not support reading
2020-04-17 07:24:54 UTC | CORE | INFO | (pkg/logs/scheduler/scheduler.go:119 in Unschedule) | New service to remove: entity: docker://c86bdf36814fd5c62036f16a03a6c1c96000bc6ff7a98e88c2f5c01b6c24dfdc
2020-04-17 07:24:54 UTC | CORE | INFO | (pkg/logs/scheduler/scheduler.go:119 in Unschedule) | New service to remove: entity: docker://45747b7373e772da8a3f2f398c0f45165d1843f937464832bb22b53b0c9ccc78
2020-04-17 07:24:55 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check cpu
2020-04-17 07:24:55 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check cpu, next runs will be logged every 500 runs
2020-04-17 07:24:55 UTC | CORE | INFO | (pkg/logs/scheduler/scheduler.go:119 in Unschedule) | New service to remove: entity: docker://bde850ddb39ee1ac9896cf62e723585cfec6d44f45a47ebc620ca1265d573773
2020-04-17 07:24:55 UTC | CORE | INFO | (pkg/logs/scheduler/scheduler.go:119 in Unschedule) | New service to remove: entity: docker://ea346b054f4d2c8abbef8bb01e7288a477e8c4f2bff99320579b6745c9716f09
2020-04-17 07:24:55 UTC | CORE | INFO | (pkg/logs/scheduler/scheduler.go:119 in Unschedule) | New service to remove: entity: docker://36c878491ce028213eb4fe07923718e6620d423416919156a61e9bc3b22c841d
2020-04-17 07:24:55 UTC | CORE | INFO | (pkg/logs/scheduler/scheduler.go:119 in Unschedule) | New service to remove: entity: docker://1f5d11d42809127e136d9a2f1e3abe8c75ab4fa926b62f5a9b3dde99a98fa1cf
2020-04-17 07:24:58 UTC | CORE | INFO | (pkg/logs/scheduler/scheduler.go:119 in Unschedule) | New service to remove: entity: docker://6e6416856180834a2ead4965625c78e8319bac789084bbc6215491d538330ada
2020-04-17 07:24:58 UTC | TRACE | INFO | (pkg/trace/info/stats.go:101 in LogStats) | No data received
2020-04-17 07:24:59 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check network
2020-04-17 07:24:59 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check network, next runs will be logged every 500 runs
2020-04-17 07:24:59 UTC | CORE | INFO | (pkg/logs/scheduler/scheduler.go:119 in Unschedule) | New service to remove: entity: docker://cb6abd4a8fd5ffbe5b2c2b1a99aab21e6dd52cbb3a011cb60cbe97f89fe16815
2020-04-17 07:25:00 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check load
2020-04-17 07:25:00 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check load, next runs will be logged every 500 runs
2020-04-17 07:25:01 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check file_handle
2020-04-17 07:25:01 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check file_handle, next runs will be logged every 500 runs
2020-04-17 07:25:02 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check disk
2020-04-17 07:25:02 UTC | CORE | WARN | (pkg/collector/python/datadog_agent.go:118 in LogMessage) | disk:e5dffb8bef24336f | (disk.py:77) | Unable to get disk metrics for /host/proc/sys/fs/binfmt_misc: [Errno 40] Too many levels of symbolic links: '/host/proc/sys/fs/binfmt_misc'
2020-04-17 07:25:02 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check disk, next runs will be logged every 500 runs
2020-04-17 07:25:06 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check uptime
2020-04-17 07:25:06 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check uptime, next runs will be logged every 500 runs
2020-04-17 07:25:07 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check memory
2020-04-17 07:25:07 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check memory, next runs will be logged every 500 runs
2020-04-17 07:25:08 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check io
2020-04-17 07:25:08 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check io, next runs will be logged every 500 runs
2020-04-17 07:25:09 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check docker
2020-04-17 07:25:09 UTC | CORE | WARN | (pkg/util/containers/metrics/cgroup_detect.go:126 in parseCgroupMountPoints) | No mountPoints were detected, current cgroup root is: /host/sys/fs/cgroup/