"Leader not known" in a single SR instance #3910
Description
We have an issue with the Schema Registry version: 7.5.0 commitId: d93bbc2bc1bb381855e1ba7182670329e81620a3
Schema Registry deployed in a single instance (no cluster setup, the SR deployed on VM). During the last rebalancing, the Schema Registry found several advertising URLs, although we have only one Schema Registry node. Then there were many attempts to join the group and it is unclear whether it succeeded. After all Schema Registry stuck in read-only state (reject new schemas).
The problem is a synced group MemberID (sr-1-2a211d04-2124-464c-8aef-49630b7a2593) differs from the MemberID of the current instance (sr-1-6df2e0ce-98b5-4917-9da6-35540a322923), so the Service Registry assume it is not a leader and all attempts to write a schema resulted in the error "Register schema request failed since leader is unknown". The Service Registry was stuck in this non-leader state and only a restart helped.
[2025-08-15 18:21:21,383] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Successfully synced group in generation Generation{generationId=6, memberId='sr-1-2a211d04-2124-464c-8aef-49630b7a2593', protocol='v0'} (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
[2025-08-15 18:21:21,383] INFO Finished rebalance with leader election result: Assignment{version=1, error=1, leader='sr-1-6df2e0ce-98b5-4917-9da6-35540a322923', leaderIdentity=version=1,host=192.168.0.100,port=8081,scheme=http,leaderEligibility=true} (io.confluent.kafka.schemaregistry.leaderelector.kafka.KafkaGroupLeaderElector)
Full log:
[2025-08-15 18:21:21,331] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Request joining group due to: group is already rebalancing (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
[2025-08-15 18:21:21,333] INFO Rebalance started (io.confluent.kafka.schemaregistry.leaderelector.kafka.KafkaGroupLeaderElector)
[2025-08-15 18:21:21,333] INFO [Schema registry clientId=sr-1, groupId=schema-registry] (Re-)joining group (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
[2025-08-15 18:21:21,346] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Successfully joined group with generation Generation{generationId=6, memberId='sr-1-2a211d04-2124-464c-8aef-49630b7a2593', protocol='v0'} (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
[2025-08-15 18:21:21,347] ERROR Found duplicate URLs for schema registry group members. This indicates a misconfiguration and is common when executing in containers. Use the host.name configuration to set each instance's advertised host name to a value that is routable from all other schema registry instances. (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
[2025-08-15 18:21:21,383] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Successfully synced group in generation Generation{generationId=6, memberId='sr-1-2a211d04-2124-464c-8aef-49630b7a2593', protocol='v0'} (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
[2025-08-15 18:21:21,383] INFO Finished rebalance with leader election result: Assignment{version=1, error=1, leader='sr-1-6df2e0ce-98b5-4917-9da6-35540a322923', leaderIdentity=version=1,host=192.168.0.100,port=8081,scheme=http,leaderEligibility=true} (io.confluent.kafka.schemaregistry.leaderelector.kafka.KafkaGroupLeaderElector)
[2025-08-15 18:21:21,394] ERROR Unexpected exception in schema registry group processing thread (io.confluent.kafka.schemaregistry.leaderelector.kafka.KafkaGroupLeaderElector)
java.lang.IllegalStateException: The schema registry group contained multiple members advertising the same URL. Verify that each instance has a unique, routable listener by setting the 'listeners' configuration. This error may happen if executing in containers where the default hostname is 'localhost'.
at io.confluent.kafka.schemaregistry.leaderelector.kafka.KafkaGroupLeaderElector.onAssigned(KafkaGroupLeaderElector.java:247)
at io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator.onJoinComplete(SchemaRegistryCoordinator.java:149)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:478)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:389)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:369)
at io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator.poll(SchemaRegistryCoordinator.java:113)
at io.confluent.kafka.schemaregistry.leaderelector.kafka.KafkaGroupLeaderElector$1.run(KafkaGroupLeaderElector.java:199)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
// ...
[2025-08-15 18:22:30,540] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Request joining group due to: group is already rebalancing (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
[2025-08-15 18:22:33,544] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Request joining group due to: group is already rebalancing (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
[2025-08-15 18:22:36,548] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Request joining group due to: group is already rebalancing (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
// ... 70 messages omitted ...
[2025-08-15 18:26:18,855] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Request joining group due to: group is already rebalancing (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
[2025-08-15 18:26:21,358] WARN [Schema registry clientId=sr-1, groupId=schema-registry] consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records. (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
[2025-08-15 18:26:21,358] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Member sr-1-2a211d04-2124-464c-8aef-49630b7a2593 sending LeaveGroup request to coordinator 192.168.0.101:9092 (id: 2147483644 rack: null) due to consumer poll timeout has expired. (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
[2025-08-15 18:26:21,358] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Resetting generation and member id due to: consumer pro-actively leaving the group (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
[2025-08-15 18:26:21,359] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Request joining group due to: consumer pro-actively leaving the group (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
[2025-08-15 22:48:03,679] INFO 192.168.0.4 - client [15/Aug/2025:19:48:03 +0000] "POST /subjects/google%2Fprotobuf%2Ftimestamp.proto?normalize=False&deleted=False HTTP/1.1" 200 563 "-" "-" 3 (io.confluent.rest-utils.requests)
[2025-08-16 01:36:04,167] INFO 192.168.0.4 - client [15/Aug/2025:22:36:04 +0000] "POST /subjects/google%2Fprotobuf%2Ftimestamp.proto?normalize=False&deleted=False HTTP/1.1" 200 563 "-" "-" 3 (io.confluent.rest-utils.requests)
[2025-09-22 23:50:37,675] INFO 192.168.0.5 - client [22/Sep/2025:20:50:37 +0000] "POST /subjects/google%2Fprotobuf%2Ftimestamp.proto?normalize=False&deleted=False HTTP/1.1" 200 563 "-" "-" 1 (io.confluent.rest-utils.requests)
[2025-09-23 00:33:37,740] INFO 192.168.0.5 - client [22/Sep/2025:21:33:37 +0000] "POST /subjects/google%2Fprotobuf%2Ftimestamp.proto?normalize=False&deleted=False HTTP/1.1" 200 563 "-" "-" 2 (io.confluent.rest-utils.requests)
[2025-09-23 00:57:17,977] INFO Registering new schema: subject google/api/http.proto, version null, id null, type PROTOBUF, schema size 728 (io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource)
[2025-09-23 00:57:18,012] ERROR Request Failed with exception (io.confluent.rest.exceptions.DebuggableExceptionMapper)
io.confluent.kafka.schemaregistry.rest.exceptions.RestUnknownLeaderException: Leader not known.
at io.confluent.kafka.schemaregistry.rest.exceptions.Errors.unknownLeaderException(Errors.java:184)
at io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource.register(SubjectVersionsResource.java:435)
at jdk.internal.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.ServletContainer.serviceImpl(ServletContainer.java:378)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:553)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:494)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:431)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at io.confluent.kafka.schemaregistry.rest.RequestIdHandler.handle(RequestIdHandler.java:51)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: io.confluent.kafka.schemaregistry.exceptions.UnknownLeaderException: Register schema request failed since leader is unknown
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.registerOrForward(KafkaSchemaRegistry.java:848)
at io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource.register(SubjectVersionsResource.java:412)
... 63 more
[2025-09-23 00:57:18,016] INFO 192.168.0.6 - client [22/Sep/2025:21:57:17 +0000] "POST /subjects/google%2Fapi%2Fhttp.proto/versions?normalize=False HTTP/1.1" 500 50 "-" "-" 40 (io.confluent.rest-utils.requests)