netdev collector failing with 'couldn't get netstats: incorrect size' since 1.4.0

[geerlingguy]

Host operating system: output of uname -a

Linux RT-AX86U-FAC0 4.1.52 #2 SMP PREEMPT Sun Jul 24 18:28:51 EDT 2022 aarch64 ASUSWRT-Merlin

node_exporter version: output of node_exporter --version

1.4.0 (TODO: Grabbing exact output momentarily...)

node_exporter command line flags

N/A

Are you running node_exporter in Docker?

No

What did you do that produced an error?

Tried starting node_exporter on my ASUS RT-AX86U router, and hitting the /metrics endpoint.

What did you expect to see?

No errors in the log output.

What did you see instead?

The following error in the log output:

ts=2022-10-06T00:48:07.144Z caller=collector.go:169 level=error msg="collector failed" name=netdev duration_seconds=0.003381482 err="couldn't get netstats: incorrect size, want: 92 or 96"

Here are the contents of net/dev:

admin@RT-AX86U-FAC0:/tmp/mnt/SANDISK/node_exporter# cat /proc/net/dev
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
spu_us_dummy:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  sit0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  imq1:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  eth7: 1140948    2880    0   21    0     0          0       238  2684974    5992    0  570    0     0       0          0
    lo:  771360    2822    0    0    0     0          0         0   771360    2822    0    0    0     0       0          0
  eth3:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
   br0: 7475066   23742    0   69    0     0          0         0 21374265   29308    0    0    0     0       0          0
 bcmsw: 48632726   94292    0   95    0     0          0      6792 53283823   99930    0    0    0     0       0          0
  ifb1:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  imq0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  eth6: 1635313    4029    0    0    0 14646          0         0  1809752    7811    5    0    0     0       0          0
  eth2:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
ip6tnl0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  ifb0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
ip6gre0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  eth5: 6249677   22374    0  106    0     0          0       970 20705170   29611    0    0    0     0       0          0
  eth1:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
bcmswlpbk0: 20705170   29611    0    0    0     0          0      1939  6239336   22279    0    0    0     0       0          0
 dpsta:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  imq2:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
spu_ds_dummy:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  eth4:     756       8    0    0    0     0          0         3   995142    3460    0    0    0     0       0          0
  eth0: 14370789   16652    0    0    0     0          0       128  3529271   11601    0    0    0     0       0          0

Downgrading to node_exporter 1.3.1 linux arm64 release seems to fix the problem (I also had no issue on older versions as well).

[SuperQ]

1.4.0 switched from parsing /proc/net/dev to using netlink. It seems like the library isn't handling your older kernel version properly.

[SuperQ]

Weird, I double-checked with the source to Linux 4.1.52. There should be 92 bytes in the rtnl_link_stats64 struct.

[jsimonetti]

Weird, I double-checked with the source to Linux 4.1.52. There should be 92 bytes in the rtnl_link_stats64 struct.

I think you meant to look at the rtnl_link_stats struct, nevertheless, that should be 92 bytes for sure.

[geerlingguy]

It's definitely not outside the realm of possibility it's a bug in the ASUSWRT-Merlin firmware... I'm less familiar with their build process, so haven't taken a stab at any further debugging on that device in particular.

[SuperQ]

@geerlingguy I've added a patch to the netlink library in order to help find out what is being returned. You should be able to patch it in and build your own binary to debug.

go get github.com/jsimonetti/rtnetlink@6ffd7597d2f777004467661b035c9d911f89b373
GOARCH=arm64 make build

[geerlingguy]

Strange... on my Mac, I can run the built binary (./node_exporter), but if I SCP it over to the router and try running it, I get:

./node_exporter: line 1: syntax error: unexpected ")"

MD5sum matches:

admin@RT-AX86U-FAC0:/tmp/mnt/SANDISK/node_exporter# md5sum node_exporter
235e8e620c51083d6a864c937effc6c9  node_exporter

 13:59:26 ~/Downloads/node_exporter 
$ md5sum node_exporter
235e8e620c51083d6a864c937effc6c9  node_exporter

Edit: Hmm... it looks like there are configs for darwin vs linux — is there a quick way I can build on macOS for linux/arm64?

[geerlingguy]

Ah, GOOS=linux GOARCH=arm64 make build did the trick.

admin@RT-AX86U-FAC0:/tmp/mnt/SANDISK/node_exporter# ./node_exporter --version
node_exporter, version 1.4.0 (branch: master, revision: b7dd00ff8b079474b080a068ce0064d66878a348)
  build user:       jgeerling@mac-studio.geerli.net
  build date:       20221006-19:10:47
  go version:       go1.19.1
  platform:         linux/arm64

And now getting:

ts=2022-10-06T19:12:03.991Z caller=collector.go:169 level=error msg="collector failed" name=netdev duration_seconds=0.011654758 err="couldn't get netstats: incorrect LinkMessage size, want: 92 or 96, got: 116"

[SuperQ]

Thanks, that helps a bunch. I guess we need to build a netlink debugging tool to see what's going on there. Very strange.

I wonder if ASUS has published their GPL patches such that we can inspect the kernel diffs.

[jsimonetti]

My guess is they use 32bit counters but with the format of the 64stats. I have not yet come across this scenario before, but (if we can find out what the extra bytes are used for) it should be trivial to fix in the unmarshalling of the LinkStat struct.

i have a feeling they copied the atruct layout of LinkStat64 but with 32bit precision instead of 64bit. Maybe to save memory?
Edit: Actually that would still only be 100 bytes, and not 116...

[geerlingguy]

Is it in here? https://github.com/RMerl/asuswrt-merlin.ng/tree/master/release/src-rt-5.02L.07p2axhnd/kernel/linux-4.1

[jsimonetti]

Yup, here it is:
https://github.com/RMerl/asuswrt-merlin.ng/blob/cc86026010d319771af97e87f7bf0025dbaebb3e/release/src-rt-5.02L.07p2axhnd/kernel/linux-4.1/include/uapi/linux/if_link.h#L18

There are additional fields added to the standard struct... thats an additional 6*4 bytes.
I'll be honest, I have no idea how to fix that, especially since they are added in an arbitrary location. :(

[SuperQ]

Hmm, the 64-bit stats, at least they appended to the end of the struct. That's slightly less painful.

[jsimonetti]

I'd be OK with loosening the length check to require at lease 92 bytes, instead of matching the exact size of either 92 or 96. However, that would still mean the statistics are all wrong, since the struct fields don't line up for the 32-bit stats.

I have no way of fixing that last part.

[jsimonetti]

Hmm, the 64-bit stats, at least they appended to the end of the struct. That's slightly less painful.

Yeah, if they would've appended to the end of the 32-bits struct, it would be fairly trivial to fix on my end. (Though you still would not be able to see those additional BCM stats)