Failed to call mallctl with metrics and jemalloc (version 1) enabled

[jellykells]

Description

My logs are full of the following warnings

2021-08-31 22:02:23,539 - synapse.metrics.jemalloc - 176 - WARNING - sentinel - Failed to read jemalloc stats.resident: Failed to call mallctl
2021-08-31 22:02:23,539 - synapse.metrics.jemalloc - 176 - WARNING - sentinel - Failed to read jemalloc stats.metadata: Failed to call mallctl

Steps to reproduce

• enable jemalloc by setting the environment variable e.g. LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1
• enable metrics collection

The server and metrics appear to still be working properly so I'm not sure what the impact of this warning is.

Version information

• Homeserver: Other

If not matrix.org:

• Version: 1.41.1

• Install method: matrix.org Debian package repository

• Platform:

OS: Ubuntu 18.04.5 LTS
Virtualized: Yes

[reivilibre]

I notice the Docker images use /usr/lib/%s-linux-gnu/libjemalloc.so.2 as their path for the jemalloc library — I notice that has version 2 at the end whilst yours has version 1.

I suspect (but can't promise to be true) that your, older, version of jemalloc doesn't support these stats.

You'll probably be missing out on jemalloc-specific metrics (#9882), but that might not be the end of the world for you.

[reivilibre]

Maybe we 'should' be only logging the warning once and then stopping, or perhaps we should check that the loaded jemalloc is of version 2 before starting the metrics for jemalloc (if this is indeed the reason).

I note your distro is out of active support now, so the best option for you may be to update at some point and install jemalloc2, sorry.

[richvdh]

we should probably also log the error code returned by mallctl.

[jellykells]

I was using libjemalloc.so.1 because that's what the brief instructions in the README mention. However I just compiled and installed the latest version of jemalloc and the warning is no longer present.

I assume the version of jemalloc provided by the Ubuntu package either doesn't support the stats or it was compiled without support.

[richvdh]

ugh, the README needs an update.

(It also claims that "it's unclear how much of an improvement it provides on Python 3.x." - we have plenty of evidence it helps a lot on 3.x.)