Missing "Endpoint" permission in ClusterRole when installing v0.18.0

[hamzabouissi]

What happened:
installed external-dns using helm chart, when I tried to see exteranl-dns why it's not syncing ingress records, the logs I found are :
"Failed to watch" err="failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:glueops-core-external-dns:external-dns\" cannot list resource \"endpoints\"

What you expected to happen:
syncing endpoints without adding additional rbac

How to reproduce it (as minimally and precisely as possible):
just install external-dns using helm chart, and check the ClusterRole binded to your serviceaccount, here is what we had:

PolicyRule:                                                                                                                                                │
│   Resources                               Non-Resource URLs  Resource Names  Verbs                                                                         │
│   ---------                               -----------------  --------------  -----                                                                         │
│   dnsendpoints.externaldns.k8s.io/status  []                 []              [*]                                                                           │
│   pods                                    []                 []              [get watch list]                                                              │
│   services                                []                 []              [get watch list]                                                              │
│   endpointslices.discovery.k8s.io         []                 []              [get watch list]                                                              │
│   ingresses.extensions                    []                 []              [get watch list]                                                              │
│   dnsendpoints.externaldns.k8s.io         []                 []              [get watch list]                                                              │
│   ingresses.networking.k8s.io             []                 []              [get watch list]                                                              │
│   nodes                                   []                 []              [list watch]   

Anything else we need to know?:
we reverted back to use v0.15.0 because ClusterRole was considering Endpoint

PolicyRule:                                                                                                                                                │
│   Resources                               Non-Resource URLs  Resource Names  Verbs                                                                         │
│   ---------                               -----------------  --------------  -----                                                                         │
│   dnsendpoints.externaldns.k8s.io/status  []                 []              [*]                                                                           │
│   endpoints                               []                 []              [get watch list]                                                              │
│   pods                                    []                 []              [get watch list]                                                              │
│   services                                []                 []              [get watch list]                                                              │
│   ingresses.extensions                    []                 []              [get watch list]                                                              │
│   dnsendpoints.externaldns.k8s.io         []                 []              [get watch list]                                                              │
│   ingresses.networking.k8s.io             []                 []              [get watch list]                                                              │
│   nodes                                   []                 []              [list watch] 

Environment:

• External-DNS version (v0.18.0):
• DNS provider: aws

[venkatamutyala]

I think this might be related: external-secrets/external-secrets#5008

[ivankatliarchuk]

Two different projects. So not related, we keep it simple, if not required - removed.

More information is required to diagnose this issue, specifically your arguments (non-Helm) and Helm version and how helm is installed with values and arguments.

Based on a first look, the error implies that EndpointsInformer is active. However, this was removed in PR #5493.

[hamzabouissi]

@ivankatliarchuk we installed it with argocd, here is our application.yaml file:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-dns
  annotations:
    argocd.argoproj.io/sync-wave: "2"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    name: "in-cluster"
    namespace: glueops-core-external-dns
  project: glueops-core
  syncPolicy:
    syncOptions:
      - CreateNamespace=true  
    automated:
      prune: true
      selfHeal: true
    retry:
      backoff:
        duration: 10s
        factor: 2
        maxDuration: 3m0s
      limit: 5
  source:
    repoURL: 'https://github.com/kubernetes-sigs/external-dns'
    path: charts/external-dns
    targetRevision: external-dns-helm-chart-1.18.0
    helm:
      parameters:
        - name: domainFilters[0]
          value: {{ .Values.captain_domain }}
        - name: policy
          value: sync
        - name: provider
          value: aws
        - name: txtPrefix
          value: "extdns-"
        - name: metrics.serviceMonitor.enabled
          value: "true"
        - name: logLevel
          value: info
        - name: image.repository
          value: "{{ .Values.base_registry }}/{{ .Values.container_images.app_external_dns.external_dns.image.repository }}"
        - name: image.tag
          value: "{{ .Values.container_images.app_external_dns.external_dns.image.tag }}"
      values: |-
        {{- toYaml .Values.glueops_node_and_tolerations | nindent 8 }}
        env:
          - name: AWS_DEFAULT_REGION
            value: {{ .Values.externalDns.aws_region }}
          - name: AWS_SECRET_ACCESS_KEY
            value: {{ .Values.externalDns.aws_secretKey }}
          - name: AWS_ACCESS_KEY_ID
            value: {{ .Values.externalDns.aws_accessKey }}
        sources:
          - ingress
          - service
          - crd
        extraArgs: 
          - "--aws-prefer-cname"
          - "--aws-api-retries=3"
        

helm version output:
version.BuildInfo{Version:"v3.17.3", GitCommit:"e4da49785aa6e6ee2b86efd5dd9e43400318262b", GitTreeState:"clean", GoVersion:"go1.23.7"}

[ivankatliarchuk]

Hi @vflaux could you spot any reasons why 0.18 still require *v1.Endpoints?

[vflaux]

Hi @hamzabouissi,
I see you're overriding the image.tag. Could you confirm you're actually running v0.18.0 ?

The version should be in the log (2nd line):

time="xxx" level=info msg="GitCommitShort=unknown, GoVersion=go1.24.4, Platform=linux/amd64, UserAgent=ExternalDNS/v20250626-v0.18.0"

[hamzabouissi]

@vflaux am not seeing the log you're referencing to, here the logs we have, we stripped not related things :

time="2025-08-02T21:21:59Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s DefaultTargets:[] GlooNamespaces:[gloo-system] SkipperRouteGroupVersion:zalando.org/v1 Sources:[ingress service crd] Namespace: AnnotationFilter: LabelFilter: IngressClassNames:[] FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false IgnoreIngressRulesSpec:false GatewayNamespace: GatewayLabelFilter: Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:aws ProviderCacheTime:0s GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s GoogleZoneVisibility: DomainFilter:[xxxxx] ExcludeDomains:[] RegexDomainFilter: RegexDomainExclusion: ZoneNameFilter:[] ZoneIDFilter:[] TargetNetFilter:[] ExcludeTargetNets:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSProfiles:[] AWSAssumeRoleExternalID: AWSBatchChangeSize:1000 AWSBatchChangeSizeBytes:32000 AWSBatchChangeSizeValues:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:true AWSZoneCacheDuration:0s AWSSDServiceCleanup:false AWSSDCreateTag:map[] AWSZoneMatchParent:false AWSDynamoDBRegion: AWSDynamoDBTable:external-dns AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: AzureActiveDirectoryAuthorityHost: AzureZonesCacheDuration:0s CloudflareProxied:false CloudflareDNSRecordsPerPage:100 CloudflareRegionKey: CoreDNSPrefix:/skydns/ AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: OCIConfigFile:/etc/kubernetes/oci.yaml OCICompartmentOCID: OCIAuthInstancePrincipal:false OCIZoneScope:GLOBAL OCIZoneCacheDuration:0s InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081/ PDNSServerID:localhost PDNSAPIKey: PDNSSkipTLSVerify:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:default TXTPrefix: TXTSuffix: TXTEncryptEnabled:false TXTEncryptAESKey: Interval:1m0s MinEventSyncInterval:5s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint: ExoscaleAPIKey: ExoscaleAPISecret: ExoscaleAPIEnvironment:api ExoscaleAPIZone:ch-gva-2 CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: ResolveServiceLoadBalancerHostname:false RFC2136Host: RFC2136Port:0 RFC2136Zone:[] RFC2136Insecure:false RFC2136GSSTSIG:false RFC2136CreatePTR:false RFC2136KerberosRealm: RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s RFC2136BatchChangeSize:50 RFC2136UseTLS:false RFC2136SkipTLSVerify:false NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A AAAA CNAME] ExcludeDNSRecordTypes:[] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTTL:0 GoDaddyOTE:false OCPRouterName: IBMCloudProxied:false IBMCloudConfigFile:/etc/kubernetes/ibmcloud.json TencentCloudConfigFile:/etc/kubernetes/tencent-cloud.json TencentCloudZoneType: PiholeServer: PiholePassword: PiholeTLSInsecureSkipVerify:false PluralCluster: PluralProvider: WebhookProviderURL:http://localhost:8888/ WebhookProviderReadTimeout:5s WebhookProviderWriteTimeout:10s WebhookServer:false TraefikDisableLegacy:false TraefikDisableNew:false NAT64Networks:[]}"
time="2025-08-02T21:21:59Z" level=info msg="Instantiating new Kubernetes client"

[ivankatliarchuk]

So which version of external-dns is actually running? Could you share kubectl get Application external-dns -o yaml?

[ivankatliarchuk]

Just remove AWS credentials

[hamzabouissi]

@ivankatliarchuk

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"
    meta.helm.sh/release-name: glueops-platform
    meta.helm.sh/release-namespace: glueops-core
  creationTimestamp: "2025-08-04T10:05:32Z"
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  generation: 12
  labels:
    app.kubernetes.io/managed-by: Helm
  name: external-dns
  namespace: glueops-core
  resourceVersion: "1693919"
  uid: fad114fa-a07d-42b2-9872-2b59a51978ef
spec:
  destination:
    name: in-cluster
    namespace: glueops-core-external-dns
  project: glueops-core
  source:
    helm:
      parameters:
      
      - name: policy
        value: sync
      - name: provider
        value: aws
      - name: txtPrefix
        value: extdns-
      - name: metrics.serviceMonitor.enabled
        value: "true"
      - name: logLevel
        value: info
      values: "nodeSelector:\n  glueops.dev/role: glueops-platform\ntolerations:\n-
        effect: NoSchedule\n  key: glueops.dev/role\n  operator: Equal\n  value: glueops-platform\n
        \ sources:\n  - ingress\n  - service\n  - crd\nextraArgs:
        \n  - \"--aws-prefer-cname\"\n  - \"--aws-api-retries=3\""
    path: charts/external-dns
    repoURL: https://github.com/kubernetes-sigs/external-dns
    targetRevision: external-dns-helm-chart-1.18.0
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    retry:
      backoff:
        duration: 10s
        factor: 2
        maxDuration: 3m0s
      limit: 5
    syncOptions:
    - CreateNamespace=true
status:
  controllerNamespace: glueops-core
  health:
    lastTransitionTime: "2025-08-04T10:05:44Z"
    status: Healthy
  history:
  - deployStartedAt: "2025-08-04T10:05:33Z"
    deployedAt: "2025-08-04T10:05:33Z"
    id: 0
    initiatedBy:
      automated: true
    revision: 1956c7e2df7807237c13de064750f003014a3bc0
    source:
      helm:
        parameters:
      
        - name: policy
          value: sync
        - name: provider
          value: aws
        - name: txtPrefix
          value: extdns-
        - name: metrics.serviceMonitor.enabled
          value: "true"
        - name: logLevel
          value: info
        values: "nodeSelector:\n  glueops.dev/role: glueops-platform\ntolerations:\n-
          effect: NoSchedule\n  key: glueops.dev/role\n  operator: Equal\n  value:
          glueops-platform\n- ingress\n  - service\n  - crd\nextraArgs: \n  - \"--aws-prefer-cname\"\n
          \ - \"--aws-api-retries=3\""
      path: charts/external-dns
      repoURL: https://github.com/kubernetes-sigs/external-dns
      targetRevision: external-dns-helm-chart-1.18.0
  operationState:
    finishedAt: "2025-08-04T10:05:33Z"
    message: successfully synced (all tasks run)
    operation:
      initiatedBy:
        automated: true
      retry:
        backoff:
          duration: 10s
          factor: 2
          maxDuration: 3m0s
        limit: 5
      sync:
        prune: true
        revision: 1956c7e2df7807237c13de064750f003014a3bc0
        syncOptions:
        - CreateNamespace=true
    phase: Succeeded
    startedAt: "2025-08-04T10:05:33Z"
    syncResult:
      resources:
      - group: ""
        hookPhase: Running
        kind: ServiceAccount
        message: serviceaccount/external-dns created
        name: external-dns
        namespace: glueops-core-external-dns
        status: Synced
        syncPhase: Sync
        version: v1
      - group: apiextensions.k8s.io
        hookPhase: Running
        kind: CustomResourceDefinition
        message: customresourcedefinition.apiextensions.k8s.io/dnsendpoints.externaldns.k8s.io
          unchanged
        name: dnsendpoints.externaldns.k8s.io
        namespace: glueops-core-external-dns
        status: Synced
        syncPhase: Sync
        version: v1
      - group: rbac.authorization.k8s.io
        hookPhase: Running
        kind: ClusterRole
        message: "clusterrole.rbac.authorization.k8s.io/external-dns reconciled. reconciliation
          required create\n\tmissing rules added:\n\t\t{Verbs:[list watch] APIGroups:[]
          Resources:[nodes] ResourceNames:[] NonResourceURLs:[]}\n\t\t{Verbs:[get
          watch list] APIGroups:[] Resources:[pods] ResourceNames:[] NonResourceURLs:[]}\n\t\t{Verbs:[get
          watch list] APIGroups:[] Resources:[services] ResourceNames:[] NonResourceURLs:[]}\n\t\t{Verbs:[get
          watch list] APIGroups:[discovery.k8s.io] Resources:[endpointslices] ResourceNames:[]
          NonResourceURLs:[]}\n\t\t{Verbs:[get watch list] APIGroups:[extensions networking.k8s.io]
          Resources:[ingresses] ResourceNames:[] NonResourceURLs:[]}\n\t\t{Verbs:[get
          watch list] APIGroups:[externaldns.k8s.io] Resources:[dnsendpoints] ResourceNames:[]
          NonResourceURLs:[]}\n\t\t{Verbs:[*] APIGroups:[externaldns.k8s.io] Resources:[dnsendpoints/status]
          ResourceNames:[] NonResourceURLs:[]}. clusterrole.rbac.authorization.k8s.io/external-dns
          configured. Warning: resource clusterroles/external-dns is missing the kubectl.kubernetes.io/last-applied-configuration
          annotation which is required by  apply.  apply should only be used on resources
          created declaratively by either  create --save-config or  apply. The missing
          annotation will be patched automatically."
        name: external-dns
        namespace: glueops-core-external-dns
        status: Synced
        syncPhase: Sync
        version: v1
      - group: rbac.authorization.k8s.io
        hookPhase: Running
        kind: ClusterRoleBinding
        message: "clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer
          reconciled. reconciliation required create\n\tmissing subjects added:\n\t\t{Kind:ServiceAccount
          APIGroup: Name:external-dns Namespace:glueops-core-external-dns}. clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer
          configured. Warning: resource clusterrolebindings/external-dns-viewer is
          missing the kubectl.kubernetes.io/last-applied-configuration annotation
          which is required by  apply.  apply should only be used on resources created
          declaratively by either  create --save-config or  apply. The missing annotation
          will be patched automatically."
        name: external-dns-viewer
        namespace: glueops-core-external-dns
        status: Synced
        syncPhase: Sync
        version: v1
      - group: ""
        hookPhase: Running
        kind: Service
        message: service/external-dns created
        name: external-dns
        namespace: glueops-core-external-dns
        status: Synced
        syncPhase: Sync
        version: v1
      - group: apps
        hookPhase: Running
        kind: Deployment
        message: deployment.apps/external-dns created
        name: external-dns
        namespace: glueops-core-external-dns
        status: Synced
        syncPhase: Sync
        version: v1
      revision: 1956c7e2df7807237c13de064750f003014a3bc0
      source:
        helm:
          parameters:
     
          - name: policy
            value: sync
          - name: provider
            value: aws
          - name: txtPrefix
            value: extdns-
          - name: metrics.serviceMonitor.enabled
            value: "true"
          - name: logLevel
            value: info
          values: "nodeSelector:\n  glueops.dev/role: glueops-platform\ntolerations:\n-
            effect: NoSchedule\n  key: glueops.dev/role\n  operator: Equal\n  value:
            glueops-platform\n sources:\n
            \ - ingress\n  - service\n  - crd\nextraArgs: \n  - \"--aws-prefer-cname\"\n
            \ - \"--aws-api-retries=3\""
        path: charts/external-dns
        repoURL: https://github.com/kubernetes-sigs/external-dns
        targetRevision: external-dns-helm-chart-1.18.0
  reconciledAt: "2025-08-04T10:05:44Z"
  resources:
  - health:
      status: Healthy
    kind: Service
    name: external-dns
    namespace: glueops-core-external-dns
    status: Synced
    version: v1
  - kind: ServiceAccount
    name: external-dns
    namespace: glueops-core-external-dns
    status: Synced
    version: v1
  - group: apiextensions.k8s.io
    kind: CustomResourceDefinition
    name: dnsendpoints.externaldns.k8s.io
    status: Synced
    version: v1
  - group: apps
    health:
      status: Healthy
    kind: Deployment
    name: external-dns
    namespace: glueops-core-external-dns
    status: Synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRole
    name: external-dns
    status: Synced
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding
    name: external-dns-viewer
    status: Synced
    version: v1
  sourceHydrator: {}
  sourceType: Helm
  summary:
    images:
    - registry.k8s.io/external-dns/external-dns:v0.18.0
  sync:
    comparedTo:
      destination:
        name: in-cluster
        namespace: glueops-core-external-dns
      source:
        helm:
          parameters:
         
          - name: policy
            value: sync
          - name: provider
            value: aws
          - name: txtPrefix
            value: extdns-
          - name: metrics.serviceMonitor.enabled
            value: "true"
          - name: logLevel
            value: info
          values: "nodeSelector:\n  glueops.dev/role: glueops-platform\ntolerations:\n-
            effect: NoSchedule\n  key: glueops.dev/role\n  operator: Equal\n  value:
            glueops-platform\n
            \ sources:\n
            \ - ingress\n  - service\n  - crd\nextraArgs: \n  - \"--aws-prefer-cname\"\n
            \ - \"--aws-api-retries=3\""
        path: charts/external-dns
        repoURL: https://github.com/kubernetes-sigs/external-dns
        targetRevision: external-dns-helm-chart-1.18.0
    revision: 1956c7e2df7807237c13de064750f003014a3bc0
    status: Synced

[ivankatliarchuk]

That's strange. I haven't been able to reproduce this on my end. Can you confirm if your ExternalDNS instance is actually running version 0.18? The only difference, I'm not using ArgoCD

[ivankatliarchuk]

Another thing, if you see no version in logs, this is the line in code

external-dns/controller/execute.go

Line 98 in 111df9f

log.Info(externaldns.Banner())

, it could be that it's an older version of a service. Worth to review environment.