(bug) high memory usage and OOMs with large payloads to /v1/data/collector and debug logging

[hairyhenderson]

Observed behaviour

I ran into this in a bit of an unexpected way... After configuring a 5s timeout on the HTTP client on my OFREP provider (using github.com/open-feature/go-sdk-contrib/providers/go-feature-flag), the payloads to /v1/data/collector started growing:

At the same time, memory usage started to grow and eventually OOM:

After some digging it appears the majority of memory usage is here:

go-feature-flag/cmd/relayproxy/api/server.go

Line 66 in 6af17ee

s.zapLog.Debug("Request info", zap.ByteString("request_body", reqBody))

This is a middleware that, if the logLevel is debug, will simply log the entire request body, regardless of size.

For reasons I'm not entirely clear on at the moment, logging these bodies results in some uncontrolled memory usage and eventually an OOM.

Expected Behavior

Stable memory usage 😉

Steps to reproduce

It's a bit complicated in practice, but in concept quite simple - I managed to reproduce this locally by using Cursor to throw large (100kB+) requests at the relayproxy. In one test, 20k requests with 500 concurrent workers managed to increase the heap by over 1000MB in a matter of minutes.

Obviously, a simple solution is to turn off debug logging; this results in quite stable memory usage, hovering around 100MB in my local setup.

I also experimented with truncating the byte slice to 8192b, which had essentially the same effect as disabling debug logging. So, I think if it makes sense that might be a good middle ground.

[hairyhenderson]

Something additional as I've been looking at traces - it's clear that client disconnects aren't resulting in cancellation of the in-process requests: