Ballast processor seems to escape virtual memory and enter RSS

[datsabk]

Component(s)

No response

What happened?

Description

Open telemetry agent goes around utilising over 6Gb of memory while scraping merely 6-7 pods on the node.

Steps to Reproduce

Use the provided configuration.

Expected Result

In agent mode, it should ideally use less than 1Gi memory to run smoothly on the agent

Actual Result

Uses over 6Gi memory at times. Have cases where it even crosses 64Gi memory if the node has that level of memory available. Looks like GC never runs

Collector version

0.69.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

exporters:
            logging: null
            prometheusremotewrite:
              endpoint: "https://aps-workspaces.us-west-2.amazonaws.com/workspaces/ws-xxx/api/v1/remote_write"
              external_labels:
                cluster_name: prodobs
              auth:
                authenticator: sigv4auth
            
          extensions:
            health_check: {}
            memory_ballast: 
              size_in_percentage: 33
            sigv4auth:
              assume_role:
                sts_region: "us-west-2"
          processors:
            batch: {}
          receivers:
            jaeger: null
            zipkin: null
            otlp:
              protocols:
                grpc:
                  endpoint: 0.0.0.0:4327
                http:
                  endpoint: 0.0.0.0:4328
            prometheus:
              config:
                scrape_configs:
                - job_name: pods-scraper
                  scrape_interval: 15s
                  kubernetes_sd_configs:
                    - role: pod
                  metric_relabel_configs:
                    - action: labeldrop
                      regex: prometheus_(.*)
                    - action: labeldrop
                      regex: beta_(.*)
                    - action: labeldrop
                      regex: (app_kubernetes_io_instance|url|pod_template_hash|prometheus|container_id)
                    - source_labels: [__name__]
                      regex: '(functions_.*|feature_.*|span_.*)'
                      action: drop
                  relabel_configs:
                    - source_labels: [__meta_kubernetes_pod_name]
                      regex: '(node-exporter|kube-state-metrics|yace|kube-dns-cache)-.*'
                      action: drop
                    - source_labels: [__meta_kubernetes_pod_node_name]
                      action: keep
                      regex: $NODE_NAME
                    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
                      separator: ;
                      regex: "true"
                      action: keep
                    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
                      separator: ;
                      regex: (.+)
                      target_label: __metrics_path__
                      replacement: $$1
                      action: replace
                    - action: replace
                      regex: ([^:]+)(?::\d+)?;(\d+)
                      replacement: $$1:$$2
                      source_labels:
                        - __address__
                        - __meta_kubernetes_pod_annotation_prometheus_io_port
                      target_label: __address__
                    - source_labels: [__meta_kubernetes_pod_container_name]
                      target_label: container
                    - action: labeldrop
                      regex: __meta_kubernetes_pod_label_k2_(.+)
                    - action: labelmap
                      regex: __meta_kubernetes_pod_label_(.+)
                    - source_labels: [__meta_kubernetes_namespace]
                      action: replace
                      target_label: kubernetes_namespace
                    - source_labels: [__meta_kubernetes_pod_name]
                      action: replace
                      target_label: pod
                    - action: drop
                      regex: '.*_kubernetes_io_.*'
                    - source_labels: [__meta_kubernetes_pod_node_name]
                      separator: ;
                      regex: (.*)
                      target_label: node
                      replacement: $$1
                      action: replace
                    - source_labels: [__meta_kubernetes_pod_label_cluster]
                      target_label: cluster
                    - source_labels: [quantile]
                      target_label: le
                    
          service:
            telemetry:
              metrics:
                address: 0.0.0.0:8888
            extensions:
              - sigv4auth
              - health_check
              - memory_ballast
            pipelines:
              logs: null
              traces: null
              metrics:
                exporters:
                  - prometheusremotewrite 
                processors:
                  - batch
                receivers:
                  - otlp
                  - prometheus

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners for receiver/prometheus: @Aneurysm9 @dashpole. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

Pinging code owners for exporter/prometheus: @Aneurysm9. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[Aneurysm9]

Are you able to enable the pprofextension and obtain a heap dump from the collector while it is using more memory than you expect?

[datsabk]

We are able to - but that didn't seem to be of much help.
In general do you see anything that could be wrong with the way configuration has been done?

[Aneurysm9]

You have not provided me with a heap dump. How have you decided that it wasn't of much help to me?

[datsabk]

@Aneurysm9 - Working on getting you a profile dump. The behavior is random. As soon as I restart, the high pod usage goes away. So waiting for it to be back in high usage state and then would provide you with the profile

[datsabk]

Here are all the profiles @Aneurysm9
allocs.txt
goroutine.txt
heap.txt
threadcreate.txt

As you can see, in the screenshot, the pod immediately spiked to 5+ GB memory on startup and then has been stable at that level. This doesn't happen everytime. Its random.

We have also added memory limiter but that doesn't seem to do much other than repeatedly calling gc and dropping samples (which could potentially be data loss)

[Aneurysm9]

Are you running this on a box with 16GB of RAM? You have the memory ballast extension configured to use 33% of system memory as ballast and it is using just over 5GB as ballast:

heap profile: 68: 5448836304 [1650: 5492618640] @ heap/1048576
1: 5443952640 [1: 5443952640] @ 0x2f1dab4 0xb714dd 0xc7e211 0xcee3cb 0xcee6a5 0xcef999 0xcaa562 0xcaaddd 0x519781d 0x5197821 0x5197758 0x51975b8 0x43cd72 0x46ef81
#       0x2f1dab3       go.opentelemetry.io/collector/extension/ballastextension.(*memoryBallast).Start+0x93    go.opentelemetry.io/collector/extension/ballastextension@v0.69.0/memory_ballast.go:49
#       0xb714dc        go.opentelemetry.io/collector/service/extensions.(*Extensions).Start+0x1bc              go.opentelemetry.io/collector@v0.69.0/service/extensions/extensions.go:46
#       0xc7e210        go.opentelemetry.io/collector/service.(*Service).Start+0x230                            go.opentelemetry.io/collector@v0.69.0/service/service.go:128
#       0xcee3ca        go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents+0x52a   go.opentelemetry.io/collector@v0.69.0/otelcol/collector.go:180
#       0xcee6a4        go.opentelemetry.io/collector/otelcol.(*Collector).Run+0x64                             go.opentelemetry.io/collector@v0.69.0/otelcol/collector.go:205
#       0xcef998        go.opentelemetry.io/collector/otelcol.NewCommand.func1+0x418                            go.opentelemetry.io/collector@v0.69.0/otelcol/command.go:53
#       0xcaa561        github.com/spf13/cobra.(*Command).execute+0x861                                         github.com/spf13/cobra@v1.6.1/command.go:916
#       0xcaaddc        github.com/spf13/cobra.(*Command).ExecuteC+0x3bc                                        github.com/spf13/cobra@v1.6.1/command.go:1044
#       0x519781c       github.com/spf13/cobra.(*Command).Execute+0x5c                                          github.com/spf13/cobra@v1.6.1/command.go:968
#       0x5197820       main.runInteractive+0x60                                                                github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:32
#       0x5197757       main.run+0x1d7                                                                          github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:11
#       0x51975b7       main.main+0x37                                                                          github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:25
#       0x43cd71        runtime.main+0x211                                                                      runtime/proc.go:250

This seems to be working as intended.

[datsabk]

Memory ballast got it under control. Without memory ballast it was reaching 12Gi if the node had that amount of memory available.

Also wouldn't memory ballast consume 33% what is available? We have about 500 other agents running, all sitting well below 2Gi - there are agents running on even larger nodes but still the consumption is not this high.

[datsabk]

@Aneurysm9 One more thing we observed today is - memory heap graph shoes high consumption for memory ballast even for the pods that are using practically low memory looking at the metrics. These pods reside on nodes which do have sufficient memory available for memory ballast to get hold of. My question here would be - How does memory ballast really function (if you are aware of it). Does it reserve the desired memory or just tries to capture until available?

[Aneurysm9]

https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap/ explains the mechanics of a memory ballast generally. Look to the implementation for details. It basically makes a big slice and then never uses it. That slice stays in virtual memory, so if you're looking at RSS you will not see the impact of the ballast there.

[datsabk]

That's what I'd expect but unfortunately not the case. That is exactly the reason why I'm confused about what's happening. To my surprise, even after setting memory ballast to exactly 1024Mib in size, some pods still show memory usage around 200-500MiB and same trend for memory.rss too

One on the left is memory.usage while one on the right is memory.rss

[datsabk]

@Aneurysm9 - Wonder if this is whats causing it to take up RSS
https://groups.google.com/g/golang-nuts/c/66d0cItfkjY

[Aneurysm9]

That's a plausible explanation. I believe we've recently discussed whether the ballast extension is even needed with the availability of GOMEMLIMIT. Are you able to set an appropriate limit and GOGC value and remove the ballast, as suggested here?