The 'Clusters' dashboard stops showing connected Gitops clusters if any connected cluster is not reachable #806
Description
- Connect at least two GitopsCluster to the management cluster. The connected clusters could all be kind , EKS or GKE.
1st cluster kind
2nd cluster EKS - Once the dashboard show both clusters as ready then bring the first (kind) cluster down.
Now the dashboard would only shows the error message.
I expect the EKS cluster to be visible as 'Ready' and the kind cluster should be shown something other than 'Ready'
cluster-controller logs:
saeed@Saeeds-MBP weave-gitops-enterprise % kubectl logs my-mccp-cluster-controller-6fdfd7d455-c6k85 -n flux-system I0520 08:55:16.231555 1 request.go:665] Waited for 1.037394276s due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/kustomize.toolkit.fluxcd.io/v1beta1?timeout=32s 1.6530369166879685e+09 INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"} 1.6530369166884162e+09 INFO setup starting manager 1.6530369166889474e+09 INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"} 1.6530369166889858e+09 INFO Starting server {"kind": "health probe", "addr": "[::]:8081"} I0520 08:55:16.790376 1 leaderelection.go:248] attempting to acquire leader lease flux-system/c352bb9a.weave.works... I0520 08:55:16.832118 1 leaderelection.go:258] successfully acquired lease flux-system/c352bb9a.weave.works 1.653036916832532e+09 INFO controller.gitopscluster Starting EventSource {"reconciler group": "gitops.weave.works", "reconciler kind": "GitopsCluster", "source": "kind source: *v1alpha1.GitopsCluster"} 1.6530369168325942e+09 INFO controller.gitopscluster Starting EventSource {"reconciler group": "gitops.weave.works", "reconciler kind": "GitopsCluster", "source": "kind source: *v1.Secret"} 1.6530369168326032e+09 INFO controller.gitopscluster Starting EventSource {"reconciler group": "gitops.weave.works", "reconciler kind": "GitopsCluster", "source": "kind source: *v1beta1.Cluster"} 1.6530369168326066e+09 INFO controller.gitopscluster Starting Controller {"reconciler group": "gitops.weave.works", "reconciler kind": "GitopsCluster"} 1.6530369168321588e+09 DEBUG events Normal {"object": {"kind":"ConfigMap","namespace":"flux-system","name":"c352bb9a.weave.works","uid":"efa1b128-0c5e-4f43-b842-f392299fd0aa","apiVersion":"v1","resourceVersion":"54609"}, "reason": "LeaderElection", "message": "my-mccp-cluster-controller-6fdfd7d455-c6k85_cc263b54-60c9-425c-94a7-4ec1fbf7cdcd became leader"} 1.653036916832945e+09 DEBUG events Normal {"object": {"kind":"Lease","namespace":"flux-system","name":"c352bb9a.weave.works","uid":"73e23efb-db34-4f88-90c2-7f328e106c23","apiVersion":"coordination.k8s.io/v1","resourceVersion":"54610"}, "reason": "LeaderElection", "message": "my-mccp-cluster-controller-6fdfd7d455-c6k85_cc263b54-60c9-425c-94a7-4ec1fbf7cdcd became leader"} 1.6530369169337902e+09 INFO controller.gitopscluster Starting workers {"reconciler group": "gitops.weave.works", "reconciler kind": "GitopsCluster", "worker count": 1} 1.653046314328965e+09 INFO controller.gitopscluster Secret found {"reconciler group": "gitops.weave.works", "reconciler kind": "GitopsCluster", "name": "kind-kind-cluster1", "namespace": "default", "secret": {"namespace": "default", "name": "kind-cluster1-kubeconfig"}} 1.6530463143350167e+09 INFO controller.gitopscluster Secret found {"reconciler group": "gitops.weave.works", "reconciler kind": "GitopsCluster", "name": "kind-kind-cluster1", "namespace": "default", "secret": {"namespace": "default", "name": "kind-cluster1-kubeconfig"}} 1.6530601727706878e+09 INFO controller.gitopscluster Secret found {"reconciler group": "gitops.weave.works", "reconciler kind": "GitopsCluster", "name": "saeed-cluster-aws", "namespace": "default", "secret": {"namespace": "default", "name": "saeed-cluster-aws-kubeconfig"}} 1.65306017277463e+09 INFO controller.gitopscluster Secret found {"reconciler group": "gitops.weave.works", "reconciler kind": "GitopsCluster", "name": "saeed-cluster-aws", "namespace": "default", "secret": {"namespace": "default", "name": "saeed-cluster-aws-kubeconfig"}}
Cluster-service logs:
for cluster kind-kind-cluster1: Get \"https://kind-cluster1-control-plane:6443/api?timeout=32s\": dial tcp: lookup kind-cluster1-control-plane on 10.96.0.10:53: no such host"} {"level":"error","ts":1653060859.9109364,"logger":"gitops","caller":"clustersmngr/factory.go:119","msg":"Failed to update namespaces","error":"failed creating client for cluster kind-kind-cluster1: Get \"https://kind-cluster1-control-plane:6443/api?timeout=32s\": dial tcp: lookup kind-cluster1-control-plane on 10.96.0.10:53: no such host"} I0520 15:34:22.436994 15 request.go:665] Waited for 1.045970854s due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/batch/v1beta1?timeout=32s {"level":"info","ts":1653060862.9020743,"logger":"gitops.multi-cluster-fetcher","caller":"fetcher/multi.go:53","msg":"Found clusters","clusters":["Default","kind-kind-cluster1","saeed-cluster-aws"]} {"level":"error","ts":1653060889.8994498,"logger":"gitops","caller":"clustersmngr/factory.go:131","msg":"failed to create clients for","clusters":[{"Name":"Default","Server":"https://10.96.0.1:443","SecretRef":"","BearerToken":"eyJhbGciOiJSUzI1NiIsImtpZCI6IlFLVkZUMWhRVjZwTTJpZDA0cEpwYlBYMldKaldvR1dUSkpTcm9MZEJrRUUifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjg0NTcyOTE5LCJpYXQiOjE2NTMwMzY5MTksImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJmbHV4LXN5c3RlbSIsInBvZCI6eyJuYW1lIjoibXktbWNjcC1jbHVzdGVyLXNlcnZpY2UtNjRjNTc0YmNiNC05bHo2ZyIsInVpZCI6IjAzM2I2YmYyLWVjYWMtNDlmOC1iOTdmLTA2ZmZiZDNiZWI4MCJ9LCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoibXktbWNjcCIsInVpZCI6ImIxNWQ0ODI3LTNkYWMtNDdjYy05NThmLWQ0M2IwMDhlNWRiYSJ9LCJ3YXJuYWZ0ZXIiOjE2NTMwNDA1MjZ9LCJuYmYiOjE2NTMwMzY5MTksInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpmbHV4LXN5c3RlbTpteS1tY2NwIn0.trbcEdYtjVcZ_RjQ6F8opQEXWNHfmF8bniB7WAHn2HFVu0ZIokNpSdlt9cut2FbbzROfjVwFHTqtaPdsxs4eihK_AO5uOsrYGCE21PcpvciDB-AgZtkddbLQ8PLM1NSxOC9ggK4vrjK3d9BZ2cwU6untuAoRPUQoyZ4r5NMTOwwvZ1OL7YE7-VNYC4W4cn3ZUKR9rxPRuG-yWVLGqm325zwJldCU3EBClMrfCILQfqOOb71SCHfSkXbUbzAXk7NV24_WAmNi3Su7XujYXv9KaYVS3Vrs3I0NcS5qoGOa2ULQbRVNFG7PGuNm92klXSU_2LatS-UVnw0vVaPKUt3k3w","TLSConfig":{"Insecure":false,"ServerName":"","CertFile":"","KeyFile":"","CAFile":"/var/run/secrets/kubernetes.io/serviceaccount/ca.crt","CertData":null,"KeyData":null,"CAData":null,"NextProtos":null}},{"Name":"kind-kind-cluster1","Server":"https://kind-cluster1-control-plane:6443","SecretRef":"","BearerToken":"eyJhbGciOiJSUzI1NiIsImtpZCI6ImJzc3B5anJnWUZrNnJtSlFoXzhzNzZ6QmhCTTl6U1R4dkdZWUwwM05TOUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImtpbmQtY2x1c3RlcjEtdG9rZW4tNDZ2dGoiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoia2luZC1jbHVzdGVyMSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjY0YTI3YmY2LThmNGYtNDQwYS05ZTE0LWIyM2JmMDg4OWI3YyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmtpbmQtY2x1c3RlcjEifQ.qN-lk4brQrimDGfMaTiMHj442psHvL4waKcCfuVKyf703qtc4mK9BLEzKPFri9209PHHfuySsIKLE5uK1SWaBXOxUIOyRGVgHkajEp24QO0yvJGU_nVgqP_u1RGXIq6T5AoedcsoSq5mYw7-PEsMAzhU2aAixqw1cTodoOVWhevxVizslVNISBfZaofPmF0VX59NgSncpouV69wPeGHOClAfWoIqOEd5oZD2MEPRu8ZXet1AYLSgFi6kHHMz2TkRsBBf7chnydF2A6AlKthLI3F-wk03PwPiC0up3wq8pTIUBMJ_bBXxRxASJi1AmhxrQuLdcdHa6AT_9j2FgQLHow","TLSConfig":{"Insecure":false,"ServerName":"","CertFile":"","KeyFile":"","CAFile":"","CertData":null,"KeyData":null,"CAData":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeU1EVXlNREE1TURjeE0xb1hEVE15TURVeE56QTVNRGN4TTFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBT0pkCnJwNXU5NVF3WjBRZXRIL1B0enBxOVZ1ZURMRE9YMW1Sdk4yL0JzQm1QWERtMXJhMVk5aHVwMzB5bW1ETTV6Ri8KMjl1YU5EWUFObU5YQlpuTmxhTjFna1pQR2I1SnA1TkR3SmRtUjhaVVlNNnJXZUtUelFPdWExOXczMm5ITEY3TApxQ3c0NXZQZS94Ylo2cmQ1UW1WY2N2MVdESnBGcHhvVkI0RjhNREI2VUNtdGlsMFlUNERkcTBVaXFQUTc0d3NXCng5bHl1NklkUUpvcjR6bW5GekxRSXk2WTZZRmlVY0VQa0dQeDVYSUQrSXRQSkpBalZQSWFqY3BGODNpcVRJcFAKSkhXNU95UlloNE1SREpVTnRqNFQxZ2pxTyt5U0xZdmVCSUJjWjQ2b1kwb09sWGZmY3ZWYTJTaUJpWGpCM2g4agpobXR6b1I5akJuWlY2UFgzb0RNQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZCRFk2MWdrZDBEM0FJQjFMekowalkxdFVjdzlNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBSWhOVFM2RGpSMGpEZ3paSVJkSgpKcnVMM2l0bTJPd1VybEUvTjFFbzFEdDRzcGhPR0orWUZzQkwyODczL3VsM0wyNk5OaFFDT1VyeGdKek5nditvClFGWWtRcFUzRFNNRExDS3VTMWZSUDNPcTBKT2NWZ2VEMUlFbjVnM25PVHRRdmc4NEhsTzBpeG1aVVNQMnVtUTIKMmV6KytEc1RmRkRIL0N2OHJpbXM2Y2ZlcjJhL3N0SjVFM1Z4S1dvWWxvbmJTaDR4Zzg1OVFaa2FWUXR2MmY4TQpHcXlRb01hSTlOTXZlY2FCTkxucThCb2VEQWNWbUZ2aG84VTFINHU3VVRSN3o1Q1BmZ3pjY25ZR1FmVG9jbnYvCm9aclBPb2tVVVd5R1FzWlR1bE5JRTJuSXJzYW1KT29oWGRRanVVcW9iWlcyd01hYkRGRGl1WHMrVlZWVFF3VEkKN25NPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==","NextProtos":null}},{"Name":"saeed-cluster-aws","Server":"https://3A0FB49354EC168D28CE8D29D51FE6CC.gr7.eu-central-1.eks.amazonaws.com","SecretRef":"","BearerToken":"eyJhbGciOiJSUzI1NiIsImtpZCI6IjU1N0J0YVZxNG5DTmNyNkZSZGt4UGt4Mk9NclZTV1ptRTBHSFM0RTNRbzgifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InNhZWVkLWNsdXN0ZXItYXdzLXRva2VuLXh2Zjc4Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InNhZWVkLWNsdXN0ZXItYXdzIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiZjE3YjRhNjMtYTZkNS00ZjI0LTkzNDktZjBkODUyMjJjZmE1Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6c2FlZWQtY2x1c3Rlci1hd3MifQ.ZmsAsMGIkWTgET1jcJ_JQGMwJ8QzZE5jiyToks8h5EbESKBeg1IgKZsz5HHYr9dUBqiSzIPPek_nxVkKaNo4UxfXLGZ2hWesa5lJ7efkn8V3NiknhIaE5J85miLwZVYAmmPNcPsJtrXOUMolv6mU1FzN2yUjB8jcC22PjNpyHBnvexS2QsyspBq6uKb4ptyI1_RBhOXZ06QMNfjMojg4Dv_b6rorRvzIzp9_taj0n-2vOOJ4eM-h6jTe6VgxIRDSFZGfZzlD_xfec4rSjr70atji8rSsI8RfrlNB_eoZlBFxA-LyWPrhSUBPfoEvsiloWYDi4VHeUXP5_JWD88ASHA","TLSConfig":{"Insecure":false,"ServerName":"","CertFile":"","KeyFile":"","CAFile":"","CertData":null,"KeyData":null,"CAData":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeU1EVXhOekE0TVRjd04xb1hEVE15TURVeE5EQTRNVGN3TjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTTdxCnNXTlYvMkVQTXF3L0hEZVA1L3R3NWVPeXpiSk5qd096dmhrbk16cTdrQ0UzaVBmMTJXZGFRVXdlV1RpYjB1TW0KdzNtaVpjNHJ3Z1lsSncydXMyMWszSlZScHU5TUw4UE9vL3RnZmJ3SldUcHhwYjBoRVJPMnFHSkM1TmNsKytlOApab08zMUZUdkdRNWFZaXpTeWZndFNNcVBRMG9CMFNVS0tad2pwcmpJblRvY2xQUm4vbmwzVFMrRzRzNnJPVVhRCllHU2tRV20zR2V6QXk1aDNhcmJWejFlT0xpWWxQeDh0VkFicTRvUEVTV2IrZHRwYWlsdEhFSXF2YXpGcVplV2kKODloWXB6UlhtTmVvMC95ck9TU012YmR0TVpoMXlYWjl4UWlLdXh3ZSswdVZlUW1xWHNwU2oyYThiVk00TUVTdAo3bUFGUEVwSmRESkV4MlRDajZVQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZMalkxUTY5RHBwNWJrZy9sWTh6SXFYdWFWRmhNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFEQTBvY3JCT2pZNGpOTzEvbWJwbmo1cjcxeXA2dnRZdFAzRm9QbEpYeDFvVy9PZFgrMwpmR2YrRzdXNUNaNnlqd1lJdjMybC96SVhJaUE3akZMYzI5ay9lRU9BajFJNHNvSE1ENmg5TUVqUlFzZEliM25WCjF2VXREb2NSZ00vTkpjaWhNcEwvOUJCcFdiMnkzSjVxSEJmWXl6dG1tWituTUZZS1JWZllpOHlDR0gvUGQrZHMKdExlYTVQT1h4MmhibGFtZ0ZneEVMYnkwQTdEU2FHY0NKay9qdE43ZVZMQXBIZDZxeHYwZjNOYXd5c3lBdmJxSQphNXlxVVpwWnU4VmJKd3pLQVlQSE5hbFh0QlExMlRacnYwZ1FadXNZQ2V4OTNCVUlucXlaNzRZNFpSYkp5WHVnClZnM3JmUW84L1VhSFlScVcxREhMOCtPZ09lSHlUTUFoMjR5aAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==","NextProtos":null}}],"error":"failed creating client for cluster kind-kind-cluster1: Get \"https://kind-cluster1-control-plane:6443/api?timeout=32s\": dial tcp: lookup kind-cluster1-control-plane on 10.96.0.10:53: no such host"} {"level":"error","ts":1653060889.8996725,"logger":"gitops","caller":"clustersmngr/factory.go:119","msg":"Failed to update namespaces","error":"failed creating client for cluster kind-kind-cluster1: Get \"https://kind-cluster1-control-plane:6443/api?timeout=32s\": dial tcp: lookup kind-cluster1-control-plane on 10.96.0.10:53: no such host"} I0520 15:34:52.416484 15 request.go:665] Waited for 1.045814938s due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/controlplane.cluster.x-k8s.io/v1beta1?timeout=32s {"level":"info","ts":1653060892.877055,"logger":"gitops.multi-cluster-fetcher","caller":"fetcher/multi.go:53","msg":"Found clusters","clusters":["Default","kind-kind-cluster1","saeed-cluster-aws"]}