identify reason for differences in NetBox device types

[mmguero]

On startup this script is used to preload the netbox device type library.

I've noticed that the number of device types seems to be slightly different on a few fresh installations of Malcolm (like, in the 2700's range but not exactly the same). Another box I have has about 2900.

We should determine the reason for the discrepancy and see if it needs to be addressed.

[mmguero]

The total number that seems to exist in the library (we remove the WatchGuard folder manually on the image build as it causes an error on import):

devicetype-library/device-types $ grep -Pr "^model:" | grep -v WatchGuard | wc -l
2936

[mmguero]

During a normal startup without any live capture, no running enrichment, etc., I have 2930 device types. We do consistently have a few errors:

netbox-1  | ... "POST /netbox/api/dcim/device-types/ HTTP/1.1" 400 47 ...
netbox-1  | ... "POST /netbox/api/dcim/device-types/ HTTP/1.1" 400 47 ...
netbox-1  | ... "POST /netbox/api/dcim/device-types/ HTTP/1.1" 400 47 ...
netbox-1  | ... "POST /netbox/api/dcim/device-types/ HTTP/1.1" 400 47 ...
netbox-1  | ... "POST /netbox/api/dcim/device-types/ HTTP/1.1" 400 47 ...
netbox-1  | ... "POST /netbox/api/dcim/device-types/ HTTP/1.1" 400 47 ...

However I don't think this is due to anything we're doing, and it is consistently reproducible. I think these are errors in the source data itself.

So seeing that there are 6 device types that got error 400s on them, and I ended up with exactly 6 less than what I counted as a "full" complement, this seems reasonable.

We also get some errors for some other records:

netbox-1  | 127.0.0.1 - - [13/Feb/2024:22:05:20 +0000] "POST /netbox/api/dcim/power-outlet-templates/ HTTP/1.1" 400 1089 "-" "python-requests/2.31.0"
netbox-1  | 127.0.0.1 - - [13/Feb/2024:22:18:34 +0000] "POST /netbox/api/dcim/module-types/ HTTP/1.1" 400 76 "-" "python-requests/2.31.0"
netbox-1  | 127.0.0.1 - - [13/Feb/2024:22:18:34 +0000] "POST /netbox/api/dcim/module-types/ HTTP/1.1" 400 76 "-" "python-requests/2.31.0"
netbox-1  | 127.0.0.1 - - [13/Feb/2024:22:18:37 +0000] "POST /netbox/api/dcim/module-types/ HTTP/1.1" 400 76 "-" "python-requests/2.31.0"

Again, I don't have an indication that this is really a problem, but is more likely an issue with the data.

I'm going to try this a few more times but if I get the same results (2930) I don't know what else we'd do about this.

[mmguero]

Because I don't think this is high priority, I'm going to bump it and get v24.02.0 out.

[mmguero]

Kamino closed and cloned this issue to cisagov/Malcolm