snmpwalk get large amount of information but snmp_exporter does not

[lZzozZl]

Hi im trying to get information from InsightPower SNMP IPv6 for UPS
i manage to set it up so when i use snmpwalk -v2c -c public XXX.XXX.XX.XX it get name of device, is test passed and other information , but when i use the snmp_exporter it does not show this info...

how can i make snmp_exporter to get the information that snmpwalk can provide

[bastischubert]

Hi,
which module did you query, just if_mib or also the other ones like hrSystem etc ?

[bknowles]

We seem to be having a similar problem with about half of our Ethernet switches in one datacenter. But the problem seems randomly distributed between Cisco and Juniper switches. I've set the scrape_interval to be 300s and the scrape_timeout to be 200s, but almost all of the failures have a very suspicious looking scan_duration of right around 20s. We are only querying if_mib.

Is there any other information that I can provide?

[bknowles]

This one is an NVIDIA InfiniBand switch which is also having problems with snmp_exporter, but works fine with snmpwalk:

ts=2024-07-12T15:10:38.537Z caller=collector.go:456 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Starting scrape"
ts=2024-07-12T15:10:38.537Z caller=gosnmp.go:86 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 msg="Getting OIDs" oids="unsupported value type"
ts=2024-07-12T15:10:38.537Z caller=collector.go:476 level=debug auth=public_v1 target=x.y.82.4 source_address= msg="Sent module to worker" module=if_mib
ts=2024-07-12T15:10:58.539Z caller=collector.go:367 level=info auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Error scraping target" err="error getting target 10.56.82.4: request timeout (after 3 retries)"
ts=2024-07-12T15:10:58.539Z caller=collector.go:460 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Finished scrape" duration_seconds=20.002069349
ts=2024-07-12T15:15:38.537Z caller=collector.go:456 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Starting scrape"
ts=2024-07-12T15:15:38.537Z caller=gosnmp.go:86 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 msg="Getting OIDs" oids="unsupported value type"
ts=2024-07-12T15:15:38.537Z caller=collector.go:476 level=debug auth=public_v1 target=x.y.82.4 source_address= msg="Sent module to worker" module=if_mib
ts=2024-07-12T15:15:58.538Z caller=collector.go:367 level=info auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Error scraping target" err="error getting target 10.56.82.4: request timeout (after 3 retries)"
ts=2024-07-12T15:15:58.538Z caller=collector.go:460 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Finished scrape" duration_seconds=20.000856488
ts=2024-07-12T15:20:38.538Z caller=collector.go:456 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Starting scrape"
ts=2024-07-12T15:20:38.538Z caller=gosnmp.go:86 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 msg="Getting OIDs" oids="unsupported value type"
ts=2024-07-12T15:20:38.538Z caller=collector.go:476 level=debug auth=public_v1 target=x.y.82.4 source_address= msg="Sent module to worker" module=if_mib
ts=2024-07-12T15:20:58.540Z caller=collector.go:367 level=info auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Error scraping target" err="error getting target 10.56.82.4: request timeout (after 3 retries)"
ts=2024-07-12T15:20:58.540Z caller=collector.go:460 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Finished scrape" duration_seconds=20.002434144
ts=2024-07-12T15:25:38.538Z caller=collector.go:456 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Starting scrape"
ts=2024-07-12T15:25:38.538Z caller=gosnmp.go:86 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 msg="Getting OIDs" oids="unsupported value type"
ts=2024-07-12T15:25:38.538Z caller=collector.go:476 level=debug auth=public_v1 target=x.y.82.4 source_address= msg="Sent module to worker" module=if_mib
ts=2024-07-12T15:25:58.541Z caller=collector.go:367 level=info auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Error scraping target" err="error getting target 10.56.82.4: request timeout (after 3 retries)"
ts=2024-07-12T15:25:58.541Z caller=collector.go:460 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Finished scrape" duration_seconds=20.002783486
ts=2024-07-12T15:30:38.537Z caller=collector.go:456 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Starting scrape"
ts=2024-07-12T15:30:38.537Z caller=gosnmp.go:86 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 msg="Getting OIDs" oids="unsupported value type"
ts=2024-07-12T15:30:38.537Z caller=collector.go:476 level=debug auth=public_v1 target=x.y.82.4 source_address= msg="Sent module to worker" module=if_mib
ts=2024-07-12T15:30:58.541Z caller=collector.go:367 level=info auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Error scraping target" err="error getting target 10.56.82.4: request timeout (after 3 retries)"
ts=2024-07-12T15:30:58.541Z caller=collector.go:460 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Finished scrape" duration_seconds=20.003760507
ts=2024-07-12T15:35:38.537Z caller=collector.go:456 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Starting scrape"
ts=2024-07-12T15:35:38.537Z caller=gosnmp.go:86 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 msg="Getting OIDs" oids="unsupported value type"
ts=2024-07-12T15:35:38.538Z caller=collector.go:476 level=debug auth=public_v1 target=x.y.82.4 source_address= msg="Sent module to worker" module=if_mib
ts=2024-07-12T15:35:58.539Z caller=collector.go:367 level=info auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Error scraping target" err="error getting target 10.56.82.4: request timeout (after 3 retries)"
ts=2024-07-12T15:35:58.539Z caller=collector.go:460 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Finished scrape" duration_seconds=20.002076088
ts=2024-07-12T15:40:38.538Z caller=collector.go:456 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Starting scrape"
ts=2024-07-12T15:40:38.538Z caller=gosnmp.go:86 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 msg="Getting OIDs" oids="unsupported value type"
ts=2024-07-12T15:40:38.538Z caller=collector.go:476 level=debug auth=public_v1 target=x.y.82.4 source_address= msg="Sent module to worker" module=if_mib
ts=2024-07-12T15:40:58.540Z caller=collector.go:367 level=info auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Error scraping target" err="error getting target 10.56.82.4: request timeout (after 3 retries)"
ts=2024-07-12T15:40:58.540Z caller=collector.go:460 level=debug auth=public_v1 target=x.y.82.4 source_address= worker=0 module=if_mib msg="Finished scrape" duration_seconds=20.002396807

[bknowles]

I'm re-running my snmpwalk tests to get the latest data.

[bknowles]

Okay, I have fresh snmpwalk data. Here's a CSV file that shows the vendor name, the respective IP address (partially redacted), and the elapsed time for doing the snmpwalk:

Vendor,IP,Elapsed
cisco,x.y.64.200,6:59.36
cisco,x.y.64.252,6:43.30
cisco,x.y.64.254,5:20.12
cisco,x.y.80.20,4:44.97
cisco,x.y.80.21,4:46.83
cisco,x.y.80.22,4:38.15
cisco,x.y.80.23,4:39.15
cisco,x.y.80.24,6:20.87
cisco,x.y.80.25,4:39.49
cisco,x.y.80.254,5:46.09
cisco,x.y.80.26,4:51.84
cisco,x.y.80.27,4:41.87
cisco,x.y.80.28,6:21.41
cisco,x.y.90.19,6:30.47
cisco,x.y.90.20,4:37.17
cisco,x.y.90.21,4:49.03
cisco,x.y.90.22,4:50.79
cisco,x.y.90.23,4:54.21
cisco,x.y.90.24,4:43.25
cisco,x.y.90.25,4:52.08
cisco,x.y.90.254,6:36.58
cisco,x.y.90.26,4:51.71
cisco,x.y.90.27,4:53.51
cisco,x.y.94.1,5:26.58
infiniband,x.y.82.1,1:47:34
infiniband,x.y.82.10,1:47:22
infiniband,x.y.82.11,1:47:31
infiniband,x.y.82.12,0:06.00
infiniband,x.y.82.13,1:47:47
infiniband,x.y.82.14,1:47:35
infiniband,x.y.82.15,0:06.01
infiniband,x.y.82.16,1:47:43
infiniband,x.y.82.17,1:47:31
infiniband,x.y.82.18,0:06.01
infiniband,x.y.82.19,1:47:19
infiniband,x.y.82.2,1:47:38
infiniband,x.y.82.20,1:47:49
infiniband,x.y.82.21,1:47:36
infiniband,x.y.82.22,0:06.01
infiniband,x.y.82.23,0:06.01
infiniband,x.y.82.24,1:47:37
infiniband,x.y.82.25,0:06.01
infiniband,x.y.82.26,1:47:45
infiniband,x.y.82.27,0:06.00
infiniband,x.y.82.28,1:47:35
infiniband,x.y.82.29,1:47:33
infiniband,x.y.82.3,0:06.01
infiniband,x.y.82.30,0:06.01
infiniband,x.y.82.31,1:47:34
infiniband,x.y.82.32,1:47:26
infiniband,x.y.82.33,1:47:20
infiniband,x.y.82.34,1:47:40
infiniband,x.y.82.35,0:06.01
infiniband,x.y.82.36,1:47:32
infiniband,x.y.82.37,1:47:25
infiniband,x.y.82.38,1:47:38
infiniband,x.y.82.39,0:06.01
infiniband,x.y.82.4,0:06.01
infiniband,x.y.82.40,1:47:23
infiniband,x.y.82.41,0:06.01
infiniband,x.y.82.42,1:47:39
infiniband,x.y.82.43,1:47:29
infiniband,x.y.82.44,1:47:11
infiniband,x.y.82.45,0:08.06
infiniband,x.y.82.46,1:44:07
infiniband,x.y.82.47,1:47:46
infiniband,x.y.82.48,0:06.01
infiniband,x.y.82.5,0:06.01
infiniband,x.y.82.6,1:47:50
infiniband,x.y.82.7,1:47:41
infiniband,x.y.82.8,1:47:14
infiniband,x.y.82.9,0:06.01
infiniband,x.y.92.1,1:47:14
infiniband,x.y.92.10,1:47:47
infiniband,x.y.92.11,1:47:15
infiniband,x.y.92.12,1:47:34
infiniband,x.y.92.13,1:47:20
infiniband,x.y.92.14,1:47:20
infiniband,x.y.92.15,1:47:27
infiniband,x.y.92.16,1:47:25
infiniband,x.y.92.17,1:47:10
infiniband,x.y.92.18,1:47:51
infiniband,x.y.92.19,0:06.01
infiniband,x.y.92.2,1:47:24
infiniband,x.y.92.20,1:47:31
infiniband,x.y.92.21,1:47:21
infiniband,x.y.92.22,1:47:51
infiniband,x.y.92.23,1:47:47
infiniband,x.y.92.24,1:47:51
infiniband,x.y.92.25,1:47:10
infiniband,x.y.92.26,1:47:18
infiniband,x.y.92.27,1:47:49
infiniband,x.y.92.28,0:06.01
infiniband,x.y.92.29,1:47:51
infiniband,x.y.92.3,1:47:40
infiniband,x.y.92.30,1:47:35
infiniband,x.y.92.31,1:47:32
infiniband,x.y.92.32,1:47:47
infiniband,x.y.92.33,1:47:38
infiniband,x.y.92.34,0:06.01
infiniband,x.y.92.35,1:47:23
infiniband,x.y.92.36,1:47:25
infiniband,x.y.92.37,1:47:22
infiniband,x.y.92.38,1:47:48
infiniband,x.y.92.39,1:47:29
infiniband,x.y.92.4,1:47:40
infiniband,x.y.92.40,1:44:10
infiniband,x.y.92.41,1:47:21
infiniband,x.y.92.42,1:47:01
infiniband,x.y.92.43,1:47:50
infiniband,x.y.92.44,0:06.01
infiniband,x.y.92.45,1:47:33
infiniband,x.y.92.46,1:44:19
infiniband,x.y.92.47,0:06.03
infiniband,x.y.92.48,1:47:48
infiniband,x.y.92.5,1:47:37
infiniband,x.y.92.6,1:47:34
infiniband,x.y.92.7,0:06.01
infiniband,x.y.92.8,1:47:43
infiniband,x.y.92.9,1:47:46
juniper,x.y.56.1,4:11.27
juniper,x.y.56.10,5:21.86
juniper,x.y.56.2,6:01.26
juniper,x.y.56.3,5:58.04
juniper,x.y.56.4,5:28.51
juniper,x.y.56.5,5:04.29
juniper,x.y.56.6,0:12.93
juniper,x.y.56.7,0:12.71
juniper,x.y.56.8,0:12.92
juniper,x.y.56.9,2:39.94
juniper,x.y.57.1,6:00.76
juniper,x.y.57.10,6:23.94
juniper,x.y.57.11,6:03.93
juniper,x.y.57.12,5:40.28
juniper,x.y.57.13,6:08.68
juniper,x.y.57.14,6:09.11
juniper,x.y.57.15,6:15.84
juniper,x.y.57.16,6:28.15
juniper,x.y.57.2,7:15.61
juniper,x.y.57.3,7:05.07
juniper,x.y.57.4,6:03.21
juniper,x.y.57.5,6:11.91
juniper,x.y.57.6,5:27.62
juniper,x.y.57.7,6:35.64
juniper,x.y.57.8,0:21.55
juniper,x.y.57.9,6:14.07

[bknowles]

You will see that none of the Cisco switches are now having problems with snmpwalk. Only four of the Juniper switches are having problems, and that's been attributed to the known OOM issues. We're working on that as a separate matter with Juniper. We also have some NVIDIA InfiniBand switches that I need to look at.

But for now, I'm most concerned about the Ethernet switches (Cisco and Juniper) that work fine with snmpwalk, but not with snmp_exporter.

[bknowles]

Okay, so the Prometheus & Grafana monitoring system was offline anyway, so I took the opportunity to update the IP addresses we're using to monitor the Juniper switches. Our network admin requested that we stop monitoring them from the management IP address, since that only tells us if the switch is up or not, but doesn't tell us if the switch is disconnected from the customer side. We can't do this for the Cisco switches, since they are by definition the OOB network, and they don't have any in-band ports that can be monitored.

So, I switched all the IP addresses for the Juniper switches from the management port (1Gbps) to the in-band port (100Gbps). I then restarted all the containers.

After a brief period, all Ethernet switches came up.

But now the NVIDIA Infiniband switches won't show up. So, I'll have to debug that as a separate problem.

Thanks!