Memory leak in calico-felix (KDD Syncer)

[tmjd]

It seems there is a memory leak in calico-felix. After leaving a cluster running for a day I checked on memory usage with top and noticed that calico-felix was reported as using 30% on my 2 nodes and 15% on the master. I restarted one of the calico-node pods (kubectl delete -n kube-system pod <pod-name>) and then checked memory usage and calico-felix reported 0.8% (2 hours later it is at 4.4%).

Expected Behavior

I would not expect 30% memory usage on a host with 4GB Ram.

Current Behavior

After a day of almost no usage calico-felix took 30% of the memory.

Possible Solution

Eh

Steps to Reproduce (for bugs)

1. Bring up K8s cluster with Kubeadm
2. Deploy Calico with http://docs.projectcalico.org/v2.5/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml and http://docs.projectcalico.org/v2.5/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
3. Wait a day
4. Check top on the hosts.

Context

Your Environment

• Calico version: v2.5.0
• Orchestrator version (e.g. kubernetes, mesos, rkt): K8s 1.7.4
• Operating System and version: Ubuntu 16.10

[tmjd]

Here is a dump from felix shortly after restarting calico-node

> go tool pprof calico-felix felix-mem.pprof 
Entering interactive mode (type "help" for commands)
(pprof) top 50
5659.07kB of 5659.07kB total (  100%)
Dropped 221 nodes (cum <= 28.29kB)
Showing top 50 nodes out of 75 (cum >= 512.14kB)
      flat  flat%   sum%        cum   cum%
 1536.34kB 27.15% 27.15%  1536.34kB 27.15%  runtime.mapassign
 1025.12kB 18.11% 45.26%  1025.12kB 18.11%  crypto/x509.parseCertificate
  528.17kB  9.33% 54.60%   528.17kB  9.33%  regexp.onePassCopy
  518.02kB  9.15% 63.75%   518.02kB  9.15%  github.com/projectcalico/felix/vendor/github.com/beorn7/perks/quantile.NewTargeted
     514kB  9.08% 72.83%  1032.02kB 18.24%  github.com/projectcalico/felix/vendor/github.com/prometheus/client_golang/prometheus.newSummary
  513.31kB  9.07% 81.90%   513.31kB  9.07%  regexp/syntax.(*compiler).compile
  512.08kB  9.05% 90.95%   512.08kB  9.05%  net.dnsReadConfig
  512.02kB  9.05%   100%   512.02kB  9.05%  regexp.makeOnePass.func1
         0     0%   100%  1025.12kB 18.11%  crypto/tls.(*Conn).Handshake
         0     0%   100%  1025.12kB 18.11%  crypto/tls.(*Conn).clientHandshake
         0     0%   100%  1025.12kB 18.11%  crypto/tls.(*clientHandshakeState).doFullHandshake
         0     0%   100%  1025.12kB 18.11%  crypto/x509.(*CertPool).AppendCertsFromPEM
         0     0%   100%  1025.12kB 18.11%  crypto/x509.(*Certificate).Verify
         0     0%   100%  1025.12kB 18.11%  crypto/x509.ParseCertificate
         0     0%   100%  1025.12kB 18.11%  crypto/x509.initSystemRoots
         0     0%   100%  1025.12kB 18.11%  crypto/x509.loadSystemRoots
         0     0%   100%  1025.12kB 18.11%  crypto/x509.systemRootsPool
         0     0%   100%   512.10kB  9.05%  encoding/json.(*decodeState).object
         0     0%   100%   512.10kB  9.05%  encoding/json.(*decodeState).unmarshal
         0     0%   100%   512.10kB  9.05%  encoding/json.(*decodeState).value
         0     0%   100%   512.10kB  9.05%  encoding/json.Unmarshal
         0     0%   100%  1025.34kB 18.12%  github.com/projectcalico/felix/calc.init
         0     0%   100%   512.02kB  9.05%  github.com/projectcalico/felix/config.init
         0     0%   100%   513.31kB  9.07%  github.com/projectcalico/felix/dispatcher.init
         0     0%   100%  1032.02kB 18.24%  github.com/projectcalico/felix/intdataplane.init
         0     0%   100%   518.02kB  9.15%  github.com/projectcalico/felix/routetable.init
         0     0%   100%   528.17kB  9.33%  github.com/projectcalico/felix/vendor/github.com/docker/distribution/reference.anchored
         0     0%   100%   528.17kB  9.33%  github.com/projectcalico/felix/vendor/github.com/docker/distribution/reference.init
         0     0%   100%   512.10kB  9.05%  github.com/projectcalico/felix/vendor/github.com/go-openapi/spec.(*Schema).UnmarshalJSON
         0     0%   100%   512.10kB  9.05%  github.com/projectcalico/felix/vendor/github.com/go-openapi/spec.MustLoadSwagger20Schema
         0     0%   100%   512.10kB  9.05%  github.com/projectcalico/felix/vendor/github.com/go-openapi/spec.Swagger20Schema
         0     0%   100%   512.10kB  9.05%  github.com/projectcalico/felix/vendor/github.com/go-openapi/spec.init
         0     0%   100%   513.31kB  9.07%  github.com/projectcalico/felix/vendor/github.com/projectcalico/libcalico-go/lib/backend/api.init
         0     0%   100%   513.31kB  9.07%  github.com/projectcalico/felix/vendor/github.com/projectcalico/libcalico-go/lib/backend/model.init
         0     0%   100%   512.02kB  9.05%  github.com/projectcalico/felix/vendor/github.com/projectcalico/libcalico-go/lib/client.init
         0     0%   100%   512.02kB  9.05%  github.com/projectcalico/felix/vendor/github.com/projectcalico/libcalico-go/lib/validator.init
         0     0%   100%   518.02kB  9.15%  github.com/projectcalico/felix/vendor/github.com/prometheus/client_golang/prometheus.(*summary).newStream
         0     0%   100%  1032.02kB 18.24%  github.com/projectcalico/felix/vendor/github.com/prometheus/client_golang/prometheus.NewSummary
         0     0%   100%   512.10kB  9.05%  github.com/projectcalico/felix/vendor/k8s.io/apimachinery/pkg/api/resource.init
         0     0%   100%   512.10kB  9.05%  github.com/projectcalico/felix/vendor/k8s.io/apimachinery/pkg/apimachinery/announced.(*GroupMetaFactory).Enable
         0     0%   100%   512.10kB  9.05%  github.com/projectcalico/felix/vendor/k8s.io/apimachinery/pkg/apimachinery/announced.(*GroupMetaFactory).RegisterAndEnable
         0     0%   100%   512.10kB  9.05%  github.com/projectcalico/felix/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.init
         0     0%   100%   512.10kB  9.05%  github.com/projectcalico/felix/vendor/k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme
         0     0%   100%   512.10kB  9.05%  github.com/projectcalico/felix/vendor/k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme-fm
         0     0%   100%   528.17kB  9.33%  github.com/projectcalico/felix/vendor/k8s.io/client-go/discovery.init
         0     0%   100%  1552.41kB 27.43%  github.com/projectcalico/felix/vendor/k8s.io/client-go/kubernetes.init
         0     0%   100%   512.14kB  9.05%  github.com/projectcalico/felix/vendor/k8s.io/client-go/kubernetes/scheme.init
         0     0%   100%   512.14kB  9.05%  github.com/projectcalico/felix/vendor/k8s.io/client-go/kubernetes/typed/apps/v1beta1.init
         0     0%   100%   528.17kB  9.33%  github.com/projectcalico/felix/vendor/k8s.io/client-go/pkg/api/v1.init
         0     0%   100%   512.14kB  9.05%  github.com/projectcalico/felix/vendor/k8s.io/client-go/pkg/apis/autoscaling/v1.init

[tmjd]

Image from the above dump

[tmjd]

42 minutes after pod start

> go tool pprof calico-felix 42-min.pprof 
Entering interactive mode (type "help" for commands)
(pprof) top 50
6171.09kB of 6171.09kB total (  100%)
Dropped 316 nodes (cum <= 30.85kB)
Showing top 50 nodes out of 84 (cum >= 512.14kB)
      flat  flat%   sum%        cum   cum%
 1536.34kB 24.90% 24.90%  1536.34kB 24.90%  runtime.mapassign
 1025.12kB 16.61% 41.51%  1025.12kB 16.61%  crypto/x509.parseCertificate
  528.17kB  8.56% 50.07%   528.17kB  8.56%  regexp.onePassCopy
  518.02kB  8.39% 58.46%   518.02kB  8.39%  github.com/projectcalico/felix/vendor/github.com/beorn7/perks/quantile.NewTargeted
     514kB  8.33% 66.79%  1032.02kB 16.72%  github.com/projectcalico/felix/vendor/github.com/prometheus/client_golang/prometheus.newSummary
  513.31kB  8.32% 75.11%   513.31kB  8.32%  regexp/syntax.(*compiler).compile
  512.08kB  8.30% 83.41%   512.08kB  8.30%  net.dnsReadConfig
  512.02kB  8.30% 91.70%   512.02kB  8.30%  regexp.makeOnePass.func1
  512.02kB  8.30%   100%   512.02kB  8.30%  runtime.rawstringtmp
         0     0%   100%  1025.12kB 16.61%  crypto/tls.(*Conn).Handshake
         0     0%   100%  1025.12kB 16.61%  crypto/tls.(*Conn).clientHandshake
         0     0%   100%  1025.12kB 16.61%  crypto/tls.(*clientHandshakeState).doFullHandshake
         0     0%   100%  1025.12kB 16.61%  crypto/x509.(*CertPool).AppendCertsFromPEM
         0     0%   100%  1025.12kB 16.61%  crypto/x509.(*Certificate).Verify
         0     0%   100%  1025.12kB 16.61%  crypto/x509.ParseCertificate
         0     0%   100%  1025.12kB 16.61%  crypto/x509.initSystemRoots
         0     0%   100%  1025.12kB 16.61%  crypto/x509.loadSystemRoots
         0     0%   100%  1025.12kB 16.61%  crypto/x509.systemRootsPool
         0     0%   100%   512.10kB  8.30%  encoding/json.(*decodeState).object
         0     0%   100%   512.10kB  8.30%  encoding/json.(*decodeState).unmarshal
         0     0%   100%   512.10kB  8.30%  encoding/json.(*decodeState).value
         0     0%   100%   512.10kB  8.30%  encoding/json.Unmarshal
         0     0%   100%  1025.34kB 16.62%  github.com/projectcalico/felix/calc.init
         0     0%   100%   512.02kB  8.30%  github.com/projectcalico/felix/config.init
         0     0%   100%   513.31kB  8.32%  github.com/projectcalico/felix/dispatcher.init
         0     0%   100%  1032.02kB 16.72%  github.com/projectcalico/felix/intdataplane.init
         0     0%   100%   518.02kB  8.39%  github.com/projectcalico/felix/routetable.init
         0     0%   100%   528.17kB  8.56%  github.com/projectcalico/felix/vendor/github.com/docker/distribution/reference.anchored
         0     0%   100%   528.17kB  8.56%  github.com/projectcalico/felix/vendor/github.com/docker/distribution/reference.init
         0     0%   100%   512.10kB  8.30%  github.com/projectcalico/felix/vendor/github.com/go-openapi/spec.(*Schema).UnmarshalJSON
         0     0%   100%   512.10kB  8.30%  github.com/projectcalico/felix/vendor/github.com/go-openapi/spec.MustLoadSwagger20Schema
         0     0%   100%   512.10kB  8.30%  github.com/projectcalico/felix/vendor/github.com/go-openapi/spec.Swagger20Schema
         0     0%   100%   512.10kB  8.30%  github.com/projectcalico/felix/vendor/github.com/go-openapi/spec.init
         0     0%   100%   513.31kB  8.32%  github.com/projectcalico/felix/vendor/github.com/projectcalico/libcalico-go/lib/backend/api.init
         0     0%   100%   512.02kB  8.30%  github.com/projectcalico/felix/vendor/github.com/projectcalico/libcalico-go/lib/backend/k8s.(*kubeSyncer).readFromKubernetesAPI
         0     0%   100%   512.02kB  8.30%  github.com/projectcalico/felix/vendor/github.com/projectcalico/libcalico-go/lib/backend/k8s.(*realKubeAPI).NamespaceWatch
         0     0%   100%   513.31kB  8.32%  github.com/projectcalico/felix/vendor/github.com/projectcalico/libcalico-go/lib/backend/model.init
         0     0%   100%   512.02kB  8.30%  github.com/projectcalico/felix/vendor/github.com/projectcalico/libcalico-go/lib/client.init
         0     0%   100%   512.02kB  8.30%  github.com/projectcalico/felix/vendor/github.com/projectcalico/libcalico-go/lib/validator.init
         0     0%   100%   518.02kB  8.39%  github.com/projectcalico/felix/vendor/github.com/prometheus/client_golang/prometheus.(*summary).newStream
         0     0%   100%  1032.02kB 16.72%  github.com/projectcalico/felix/vendor/github.com/prometheus/client_golang/prometheus.NewSummary
         0     0%   100%   512.10kB  8.30%  github.com/projectcalico/felix/vendor/k8s.io/apimachinery/pkg/api/resource.init
         0     0%   100%   512.10kB  8.30%  github.com/projectcalico/felix/vendor/k8s.io/apimachinery/pkg/apimachinery/announced.(*GroupMetaFactory).Enable
         0     0%   100%   512.10kB  8.30%  github.com/projectcalico/felix/vendor/k8s.io/apimachinery/pkg/apimachinery/announced.(*GroupMetaFactory).RegisterAndEnable
         0     0%   100%   512.10kB  8.30%  github.com/projectcalico/felix/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.init
         0     0%   100%   512.10kB  8.30%  github.com/projectcalico/felix/vendor/k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme
         0     0%   100%   512.10kB  8.30%  github.com/projectcalico/felix/vendor/k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme-fm
         0     0%   100%   528.17kB  8.56%  github.com/projectcalico/felix/vendor/k8s.io/client-go/discovery.init
         0     0%   100%  1552.41kB 25.16%  github.com/projectcalico/felix/vendor/k8s.io/client-go/kubernetes.init
         0     0%   100%   512.14kB  8.30%  github.com/projectcalico/felix/vendor/k8s.io/client-go/kubernetes/scheme.init

[tmjd]

A 10 minute section of logs from calico-node on the master that was at 16% memory
last-10m.txt

[tmjd]

Note: There was a manifest change that was made post 2.5.0 release that may have addressed this issue. The memory leak was noticed with the original 2.5.0 release manifest, the manifest was re-downloaded (to set up to dump the memory) and the new version was retrieved. This may be the explanation if these dumps do not show a leak.

At 2 hours
2h.txt

[tmjd]

Here are samples of top showing the memory usage observed to indicate the leak. These were from when it was first noticed (before using the updated manifest).

master

ubuntu@calico-master-0:~$ top

top - 21:54:54 up 1 day,  8:09,  1 user,  load average: 0.13, 0.55, 0.59
Tasks: 159 total,   1 running, 158 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.6 us,  2.3 sy,  0.0 ni, 89.8 id,  2.3 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  7658816 total,   124800 free,  2283532 used,  5250484 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  5059368 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                       
 5386 root      20   0 1363300 1.223g  20492 S   4.5 16.7 100:54.46 calico-felix
 4746 root      20   0  787292 522936  56400 S   8.0  6.8 248:22.74 kube-apiserver
 4230 root      20   0 1202864 104084  54596 S   2.7  1.4  76:16.22 kubelet
 4008 root      20   0 1461236  92076  36232 S   0.9  1.2  22:58.70 dockerd
 4613 root      20   0  114416  54132  30224 S   1.8  0.7  45:49.08 kube-controller
 4685 root      20   0 10.056g  42876  11560 S   3.6  0.6 113:12.75 etcd
 5061 root      20   0   54760  36340  27572 S   0.0  0.5   4:23.55 kube-proxy
 5385 root      20   0   42860  31884  20860 S   0.0  0.4  11:18.57 confd
 4556 root      20   0   59228  30976  19532 S   0.0  0.4  15:36.61 kube-scheduler
 5878 nobody    20   0   35772  25260  15072 S   0.9  0.3   2:37.49 sidecar

node

ubuntu@calico-0:~$ top

top - 15:30:40 up 1 day,  1:42,  1 user,  load average: 0.12, 0.18, 0.17
Tasks: 124 total,   1 running, 123 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.5 us,  2.0 sy,  0.0 ni, 92.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  3788240 total,   147784 free,  1458844 used,  2181612 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  2066152 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                                                                        
 4458 root      20   0 1187456 1.115g  13296 S  4.4 30.9  68:49.42 calico-felix
 3979 root      20   0  787812  92480  53408 S  1.5  2.4  33:54.19 kubelet
 3754 root      20   0  795216  78504  36460 S  2.4  2.1  12:03.99 dockerd
 4459 root      20   0   41288  29640  19468 S  0.0  0.8   9:29.02 confd
 4204 root      20   0   53444  28140  18700 S  0.0  0.7   2:59.30 kube-proxy
 1424 root      20   0  266524  22060  12744 S  0.0  0.6   0:00.14 snapd
 1698 root      20   0   66444  19360   4836 S  0.0  0.5   0:03.63 google_accounts
 1704 root      20   0   66420  19104   4680 S  0.0  0.5   0:03.67 google_ip_forwa
 1705 root      20   0   66416  19032   4660 S  0.0  0.5   0:03.66 google_clock_sk
 3774 root      20   0  328164  18848   9584 S  0.0  0.5   0:01.79 containerd
  650 root      20   0   49148   9940   5276 S  0.0  0.3   0:00.61 systemd-journal

[bcreane]

I'm seeing flat memory usage with CRD manifest changes over a 24-hour period. Deleting one of the CRDs, e.g. kubectl delete crd globalnetworkpolicies.crd.projectcalico.org provokes a ~2.5MB/minute increase in calico-felix memory usage over time.

Restoring the CRD results in flat or diminishing (~330KB/minute) calico-felix memory usage.

[fasaxc]

I think we tracked this down to failing to correctly shut down the watchers and drain channels if we fail in the middle of creating the watchers (For example, because the CRDs weren't defined.)