SIGSEGV with old kernel

[Pasha49]

Host operating system: output of uname -a

2.6.32-5-amd64 #1 SMP Mon Sep 23 22:14:43 UTC 2013 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.6.1 (branch: HEAD, revision: 4a1b776)
build user: root@586879db11e5
build date: 20230717-12:10:52
go version: go1.20.6
platform: linux/amd64
tags: netgo osusergo static_build

node_exporter command line flags

./node_exporter

node_exporter log output

ts=2023-07-24T12:47:06.629Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100
ts=2023-07-24T12:47:06.629Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xa17ae7]

goroutine 59 [running]:
github.com/prometheus/node_exporter/collector.parseNetlinkStats({0xc0001fd180, 0x9, 0x0?}, 0x0?, {0xce3460, 0xc0000b4a00})
<------>/app/collector/netdev_linux.go:69 +0x227
github.com/prometheus/node_exporter/collector.netlinkStats(0x0?, {0xce3460, 0xc0000b4a00})
<------>/app/collector/netdev_linux.go:52 +0xea
github.com/prometheus/node_exporter/collector.getNetDevStats(0x0?, {0xce3460?, 0xc0000b4a00?})
<------>/app/collector/netdev_linux.go:35 +0x2f
github.com/prometheus/node_exporter/collector.(*netDevCollector).Update(0xc0000aee80, 0x0?)
<------>/app/collector/netdev_common.go:113 +0x50
github.com/prometheus/node_exporter/collector.execute({0xbcc939, 0x6}, {0xce3ce0, 0xc0000aee80}, 0x0?, {0xce3500, 0xc000242580})
<------>/app/collector/collector.go:161 +0x9c
github.com/prometheus/node_exporter/collector.NodeCollector.Collect.func1({0xbcc939?, 0x0?}, {0xce3ce0?, 0xc0000aee80?})
<------>/app/collector/collector.go:152 +0x3d
created by github.com/prometheus/node_exporter/collector.NodeCollector.Collect
<------>/app/collector/collector.go:151 +0xd0

Are you running node_exporter in Docker?

No

What did you do that produced an error?

Just run

What did you expect to see?

What did you see instead?

[SuperQ]

Looks like this is related to the new netlink-based collectors. I'm guessing that kernel's netlink is too old to work. Try with these flags: --no-collector.netclass.netlink --no-collector.netdev.netlink.

[Pasha49]

Looks like this is related to the new netlink-based collectors. I'm guessing that kernel's netlink is too old to work. Try with these flags: --no-collector.netclass.netlink --no-collector.netdev.netlink.

It works! Thank you!

[SuperQ]

I'm going to leave this open, we should probably figure out where the nil is coming from and make sure we avoid the crash.

CC @BenoitKnecht

[dswarbrick]

Essentially a duplicate of #2641.

[dswarbrick]

@SuperQ My guess is that it's because collector/netdev_linux.go refers to msg.Attributes.Stats64

func parseNetlinkStats(links []rtnetlink.LinkMessage, filter *deviceFilter, logger log.Logger) netDevStats {
    metrics := netDevStats{}

    for _, msg := range links {
        name := msg.Attributes.Name
        stats := msg.Attributes.Stats64

        if filter.ignored(name) {
            level.Debug(logger).Log("msg", "Ignoring device", "device", name)
            continue
        }   

        // https://github.com/torvalds/linux/blob/master/include/uapi/linux/if_link.h#L42-L246
        metrics[name] = map[string]uint64{
            "receive_packets":  stats.RXPackets,   // <- panic
...

... but rtnl_link_stats64 did not exist until kernel 2.6.35. Prior to that, only the 32-bit rtnl_link_stats existed. I expect that msg.Attributes.Stats64 is nil on pre-2.6.35 kernels.

cf. https://pkg.go.dev/github.com/jsimonetti/rtnetlink#LinkAttributes

This should be a trivial fix, i.e. check whether msg.Attributes.Stats64 is nil, and if so, assign the 32-bit msg.Attributes.Stats to the stats var instead. Unfortunately it also means a lot of uint64 typecasts will be needed in the map assignments.

[SuperQ]

@dswarbrick Thanks for digging into that. The other thing we could do is just fail the output if Stat64 is nil.

[BenoitKnecht]

@dswarbrick @SuperQ Sorry, I had started looking into this some time ago but never got around to finishing it or posting about it here. Basically, I did

diff --git a/collector/netdev_linux.go b/collector/netdev_linux.go
index 3156d64..91f6bee 100644
--- a/collector/netdev_linux.go
+++ b/collector/netdev_linux.go
@@ -56,8 +56,41 @@ func parseNetlinkStats(links []rtnetlink.LinkMessage, filter *deviceFilter, logg
        metrics := netDevStats{}
 
        for _, msg := range links {
+               if msg.Attributes == nil {
+                       level.Debug(logger).Log("msg", "No netlink attributes, skipping")
+                       continue
+               }
                name := msg.Attributes.Name
                stats := msg.Attributes.Stats64
+               if stats32 := msg.Attributes.Stats; stats == nil && stats32 != nil {
+                       stats = &rtnetlink.LinkStats64{
+                               RXPackets:          uint64(stats32.RXPackets),
+                               TXPackets:          uint64(stats32.TXPackets),
+                               RXBytes:            uint64(stats32.RXBytes),
+                               TXBytes:            uint64(stats32.TXBytes),
+                               RXErrors:           uint64(stats32.RXErrors),
+                               TXErrors:           uint64(stats32.TXErrors),
+                               RXDropped:          uint64(stats32.RXDropped),
+                               TXDropped:          uint64(stats32.TXDropped),
+                               Multicast:          uint64(stats32.Multicast),
+                               Collisions:         uint64(stats32.Collisions),
+                               RXLengthErrors:     uint64(stats32.RXLengthErrors),
+                               RXOverErrors:       uint64(stats32.RXOverErrors),
+                               RXCRCErrors:        uint64(stats32.RXCRCErrors),
+                               RXFrameErrors:      uint64(stats32.RXFrameErrors),
+                               RXFIFOErrors:       uint64(stats32.RXFIFOErrors),
+                               RXMissedErrors:     uint64(stats32.RXMissedErrors),
+                               TXAbortedErrors:    uint64(stats32.TXAbortedErrors),
+                               TXCarrierErrors:    uint64(stats32.TXCarrierErrors),
+                               TXFIFOErrors:       uint64(stats32.TXFIFOErrors),
+                               TXHeartbeatErrors:  uint64(stats32.TXHeartbeatErrors),
+                               TXWindowErrors:     uint64(stats32.TXWindowErrors),
+                               RXCompressed:       uint64(stats32.RXCompressed),
+                               TXCompressed:       uint64(stats32.TXCompressed),
+                               RXNoHandler:        uint64(stats32.RXNoHandler),
+                               RXOtherhostDropped: 0,
+                       }
+               }
 
                if filter.ignored(name) {
                        level.Debug(logger).Log("msg", "Ignoring device", "device", name)

As you can see, RXOtherhostDropped is hardcoded to zero because it doesn't exist in msg.Attributes.Stats for some reason. So technically we shouldn't return that metric at all in this case, rather than return it with a value of zero, but that makes the code quite a bit more verbose.

Any opinions? Do we care that much about this? (Edit: By this I mean returning that metric with a constant value, not the panic reported in this issue, of course. 🙂)

[SuperQ]

That seems like a reasonable patch to me.