rd_refcnt_sub0: Assertion `!*"refcnt sub-zero"' failed - when the subscribed topic is deleted

[dshivashankar1994]

I am using a cpp version of librdkafka consumer (version 1.4.2)

I am subscribe to 2 topics that are existing. They got assigned fine and was receiving messages.
Now when I delete one of the topic, I was expecting 2 things
(1) consumer.consume to throw the UNKNOWN_TOPIC_OR_PART error
(2) Rebalance cb to revoke that partition

But both are not happening and instead I get the assertion error.

Logs in debug mode:
Topics are team.topic.test.abc-1 and team.topic.test.abc-2.
Deleted topic - team.topic.test.abc-1

[66536 (139930824054528) - DEBUG 20200703 20:49:47.969 IST KafkaReader.C#event_cb:184] 7, facility OFFSET, str [thrd:main]: Topic team.topic.test.abc-1 [0]: stored offset 10, committed offset 10: not including in commit
[66536 (139930824054528) - DEBUG 20200703 20:49:47.969 IST KafkaReader.C#event_cb:184] 7, facility OFFSET, str [thrd:main]: Topic team.topic.test.abc-2 [0]: stored offset 20, committed offset 20: not including in commit
[66536 (139930824054528) - DEBUG 20200703 20:49:53.245 IST KafkaReader.C#event_cb:184] 7, facility OFFSET, str [thrd:main]: Topic team.topic.test.abc-1 [0]: stored offset 10, committed offset 10: not including in commit
[66536 (139930824054528) - DEBUG 20200703 20:49:53.245 IST KafkaReader.C#event_cb:184] 7, facility OFFSET, str [thrd:main]: Topic team.topic.test.abc-2 [0]: stored offset 20, committed offset 20: not including in commit
[66536 (139930790483712) - DEBUG 20200703 20:49:55.302 IST KafkaReader.C#event_cb:184] 7, facility BROKERUA, str [thrd:sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/bootstra]: team.topic.test.abc-1 [0]: broker unavailable: fetch: Broker: Unknown topic or partition
[66536 (139930790483712) - DEBUG 20200703 20:49:55.302 IST KafkaReader.C#event_cb:184] 7, facility FASTQUERY, str [thrd:sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/bootstra]: Starting fast leader query
[66536 (139930790483712) - DEBUG 20200703 20:49:55.302 IST KafkaReader.C#event_cb:184] 7, facility FETCHADD, str [thrd:sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/bootstra]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: Removed team.topic.test.abc-1 [0] from fetch list (0 entries, opv 2): fetch backed off
[66536 (139930824054528) - DEBUG 20200703 20:49:55.746 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
[66536 (139930824054528) - DEBUG 20200703 20:49:55.746 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: Request metadata for 1 topic(s): partition leader query
[66536 (139930790483712) - DEBUG 20200703 20:49:55.803 IST KafkaReader.C#event_cb:184] 7, facility FETCHADD, str [thrd:sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/bootstra]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: Added team.topic.test.abc-1 [0] to fetch list (1 entries, opv 2, 0 messages queued): fetchable
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: ===== Received metadata (for 1 requested topics): partition leader query =====
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #1/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #2/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility STATE, str [thrd:main]: Topic team.topic.test.abc-1 changed state exists -> notexists
[66536 (139930824054528) - INFO 20200703 20:49:55.951 IST KafkaReader.C#event_cb:204] 5, facility PARTCNT, str [thrd:main]: Topic team.topic.test.abc-1 partition count changed from 1 to 0
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility REMOVE, str [thrd:main]: team.topic.test.abc-1 [0] no longer reported in metadata
[66536 (139930824054528) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility DESIRED, str [thrd:main]: Topic team.topic.test.abc-1 [0] is desired but no longer known: moving back on desired list
[66536 (139930824054528) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility BRKDELGT, str [thrd:main]: team.topic.test.abc-1 [0]: delegate to broker (none) (rktp 0x7f44080a8150, term 0, ref 7)
[66536 (139930824054528) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility BRKDELGT, str [thrd:main]: team.topic.test.abc-1 [0]: no longer delegated to broker sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3
[66536 (139930824054528) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility BRKDELGT, str [thrd:main]: team.topic.test.abc-1 [0]: delegating to broker :0/internal for partition with 0 messages (0 bytes) queued
[66536 (139930824054528) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility BRKMIGR, str [thrd:main]: Migrating topic team.topic.test.abc-1 [0] 0x7f44080a8150 from sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3 to :0/internal (sending PARTITION_LEAVE to sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3)
[66536 (139930824054528) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: 1/1 requested topic(s) seen in metadata
[66536 (139930790483712) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility FETCHADD, str [thrd:sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/bootstra]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: Removed team.topic.test.abc-1 [0] from fetch list (0 entries, opv 2): forced removal
[66536 (139930790483712) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility TOPBRK, str [thrd:sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/bootstra]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: Topic team.topic.test.abc-1 [0]: leaving broker (0 messages in xmitq, next broker :0/internal, rktp 0x7f44080a8150)
[66536 (139930790483712) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility BROADCAST, str [thrd:sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/bootstra]: Broadcasting state change
[66536 (139930815661824) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility TOPBRK, str [thrd::0/internal]: :0/internal: Topic team.topic.test.abc-1 [0]: joining broker (rktp 0x7f44080a8150, 0 message(s) queued)
[66536 (139930815661824) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility BROADCAST, str [thrd::0/internal]: Broadcasting state change
[66536 (139930815661824) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility FETCHADD, str [thrd::0/internal]: :0/internal: Added team.topic.test.abc-1 [0] to fetch list (1 entries, opv 2, 0 messages queued): fetchable
[66536 (139930790483712) - DEBUG 20200703 20:49:56.155 IST KafkaReader.C#event_cb:184] 7, facility UNKTOPIC, str [thrd:sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/bootstra]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: Received Fetch response (error 3) for unknown topic team.topic.test.abc-1 [0]: ignoring
[66536 (139930824054528) - DEBUG 20200703 20:49:56.245 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:49:56.245 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:49:56.245 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #1/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #2/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:49:57.245 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:49:57.245 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:49:57.245 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:49:58.245 IST KafkaReader.C#event_cb:184] 7, facility OFFSET, str [thrd:main]: Topic team.topic.test.abc-1 [0]: stored offset 10, committed offset 10: not including in commit
[66536 (139930824054528) - DEBUG 20200703 20:49:58.245 IST KafkaReader.C#event_cb:184] 7, facility OFFSET, str [thrd:main]: Topic team.topic.test.abc-2 [0]: stored offset 20, committed offset 20: not including in commit
[66536 (139930824054528) - DEBUG 20200703 20:49:58.246 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:49:58.246 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:49:58.246 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:49:59.246 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:49:59.246 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:49:59.246 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:50:00.246 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:50:00.246 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:50:00.246 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:50:01.246 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:50:01.246 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:50:01.246 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:50:01.526 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
...
...
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:52:50.808 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:52:50.808 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:52:50.808 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:52:51.029 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:52:51.808 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:52:51.808 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:52:51.809 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:52:52.809 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:52:52.809 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:52:52.809 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:52:52.967 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 2/2 topics: periodic topic and broker list refresh
[66536 (139930824054528) - DEBUG 20200703 20:52:52.967 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 2 topic(s): periodic topic and broker list refresh
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 2 requested topics): periodic topic and broker list refresh =====
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 2 topics
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/2: team.topic.test.abc-2 with 1 partitions
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]:   Topic team.topic.test.abc-2 partition 0 Leader 1
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #1/2: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 2/2 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility TOPICERR, str [thrd:main]: Subscribed topic not available: team.topic.test.abc-1: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility SUBSCRIPTION, str [thrd:main]: Group "team.subscriber.cpp.topic.jsh-abc": effective subscription list changed from 2 to 1 topic(s):
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility SUBSCRIPTION, str [thrd:main]:  Topic team.topic.test.abc-2 with 1 partition(s)
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility REJOIN, str [thrd:main]: Group "team.subscriber.cpp.topic.jsh-abc": subscription updated from metadata change: rejoining group
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility PAUSE, str [thrd:main]: Library pausing 2 partition(s)
console_team_subscriber: rd.h:326: rd_refcnt_sub0: Assertion `!*"refcnt sub-zero"' failed.
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility BARRIER, str [thrd:main]: team.topic.test.abc-1 [0]: rd_kafka_toppar_op_pause_resume:2383: new version barrier v3
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility PAUSE, str [thrd:main]: Pause team.topic.test.abc-1 [0] (v3)
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility BARRIER, str [thrd:main]: team.topic.test.abc-2 [0]: rd_kafka_toppar_op_pause_resume:2383: new version barrier v3
Aborted (core dumped)

[dshivashankar1994]

#0 0x00007fb188359387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007fb18835aa78 in __GI_abort () at abort.c:90
#2 0x00007fb1883521a6 in __assert_fail_base (fmt=0x7fb1884adce0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0x7fb187a0a985 "!"refcnt sub-zero"", file=file@entry=0x7fb187a0a980 "rd.h", line=line@entry=326,
function=function@entry=0x7fb187a13d39 <PRETTY_FUNCTION.5115> "rd_refcnt_sub0") at assert.c:92
#3 0x00007fb188352252 in __GI___assert_fail (assertion=assertion@entry=0x7fb187a0a985 "!"refcnt sub-zero"", file=file@entry=0x7fb187a0a980 "rd.h",
line=line@entry=326, function=function@entry=0x7fb187a13d39 <PRETTY_FUNCTION.5115> "rd_refcnt_sub0") at assert.c:101
#4 0x00007fb187920646 in rd_refcnt_sub0 (R=0x7fb1780a8310) at rd.h:326
#5 0x00007fb18794d0ff in rd_refcnt_sub0 (R=0x7fb1780a8310) at rdkafka_topic.h:171
#6 rd_kafka_topic_destroy_app (app_rkt=0x7fb1780a8260) at rdkafka_topic.c:84
#7 rd_kafka_topic_destroy (app_rkt=0x7fb1780a8260) at rdkafka_topic.c:129
#8 0x00007fb18794ef1a in rd_kafka_topic_opaque (app_rkt=) at rdkafka_topic.c:1475
#9 0x00007fb1876ff7db in RdKafka::MessageImpl::MessageImpl (rkmessage=0x7fb1780a9078, this=0x7fb154000a30) at rdkafkacpp_int.h:345
#10 RdKafka::KafkaConsumerImpl::consume (this=, timeout_ms=) at KafkaConsumerImpl.cpp:119
#11 0x00007fb189d10c9d in des::kafka::KafkaReader::poll (this=0x10000c0, consumerTimeout=1000) at KafkaReader.C:491
#12 0x00007fb189d26acf in execute_native_thread_routine () from /u/dhamodha/fps/lib/Linux.RHEL7.x86_64/libdesfps1.5.so
#13 0x00007fb1886f8ea5 in start_thread (arg=0x7fb1673c1700) at pthread_create.c:307
#14 0x00007fb1884218dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

[dshivashankar1994]

Any updates on this @edenhill ? Is there anything else I can provide to you that would help?

[edenhill]

Make sure that you are destroying each Message and Topic object exactly once.

[dshivashankar1994]

I am creating TopicPartition using RdKafka::TopicPartition::create(topic->topic(), partition->id()). But not clearing it at all.
Also I had a delete message. Even after commenting out, I face the problem.

[edenhill]

Can you share your code?

[dshivashankar1994]

I'll give you a minimal reproducer by tomorrow.

[dshivashankar1994]

I am able to reproduce this using th rdkafka_complex_example_cpp itself. After the topic gets deleted, I was getting the error topic does not exist. As soon as the topic metadata refresh happens, I am getting the assertion error.

/codemill/dhamodha/librdkafka-master/examples/rdkafka_complex_consumer_example_cpp -b mwkafka-staging-01.nyc.deshaw.com:9092,mwkafka-staging-02.nyc.deshaw.com:9092,mwkafka-staging-01.dr.deshaw.com:9092,mwkafka-staging-02.dr.deshaw.com:9092,mwkafka-staging-01.tbd.deshaw.com:909 -g new_cons_gp -X security.protocol=SASL_PLAINTEXT -X sasl.kerberos.service.name=kafka -X sasl.mechanism=GSSAPI -X sasl.kerberos.kinit.cmd='skinit --force' -X client.id=cppcon fps.dhamodha.test_hello 
dhamodha acquired ticket: client dhamodha/hostbased@UNIX.DESHAW.COM, server krbtgt/UNIX.DESHAW.COM@UNIX.DESHAW.COM
% Created consumer cppcon#consumer-1
2020-07-07 10:04:45.922: LOG-3-FAIL: [thrd:sasl_plaintext://mwkafka-staging-01.tbd.deshaw.com:909/bootstra]: sasl_plaintext://mwkafka-staging-01.tbd.deshaw.com:909/bootstrap: Connect to ipv4#10.236.30.7:909 failed: Connection refused (after 206ms in state CONNECT)
2020-07-07 10:04:45.922: ERROR (Local: Broker transport failure): sasl_plaintext://mwkafka-staging-01.tbd.deshaw.com:909/bootstrap: Connect to ipv4#10.236.30.7:909 failed: Connection refused (after 206ms in state CONNECT)
RebalanceCb: Local: Assign partitions: fps.dhamodha.test_hello[0], 
2020-07-07 10:05:41.361: LOG-5-PARTCNT: [thrd:main]: Topic fps.dhamodha.test_hello partition count changed from 1 to 0
2020-07-07 10:05:41.362: ERROR (Local: Unknown partition): fps.dhamodha.test_hello [0]: desired partition no longer exists (Local: Unknown partition)
2020-07-07 10:05:41.362: ERROR (Broker: Unknown topic or partition): fps.dhamodha.test_hello [0]: topic does not exist (Broker: Unknown topic or partition)
2020-07-07 10:05:41.921: ERROR (Broker: Unknown topic or partition): fps.dhamodha.test_hello [0]: topic does not exist (Broker: Unknown topic or partition)
...
...
2020-07-07 10:09:44.920: ERROR (Broker: Unknown topic or partition): fps.dhamodha.test_hello [0]: topic does not exist (Broker: Unknown topic or partition)
rdkafka_complex_consumer_example_cpp: rd.h:326: rd_refcnt_sub0: Assertion `!*"refcnt sub-zero"' failed.
Aborted (core dumped)

From core:

Core was generated by `/codemill/dhamodha/librdkafka-master/examples/rdkafka_complex_consumer_example_'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f95a794b387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
55	  return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
[Current thread is 1 (Thread 0x7f95a9cc07c0 (LWP 38127))]
(gdb) bt
#0  0x00007f95a794b387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007f95a794ca78 in __GI_abort () at abort.c:90
#2  0x00007f95a79441a6 in __assert_fail_base (fmt=0x7f95a7a9fce0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x4ee839 "!*\"refcnt sub-zero\"", 
    file=file@entry=0x4ee834 "rd.h", line=line@entry=326, function=function@entry=0x4f7b59 <__PRETTY_FUNCTION__.5115> "rd_refcnt_sub0") at assert.c:92
#3  0x00007f95a7944252 in __GI___assert_fail (assertion=assertion@entry=0x4ee839 "!*\"refcnt sub-zero\"", file=file@entry=0x4ee834 "rd.h", line=line@entry=326, 
    function=function@entry=0x4f7b59 <__PRETTY_FUNCTION__.5115> "rd_refcnt_sub0") at assert.c:101
#4  0x00000000004086cb in rd_refcnt_sub0 (R=0x7f959c0026f0) at rd.h:326
#5  0x000000000043efaf in rd_refcnt_sub0 (R=0x7f959c0026f0) at rdkafka_topic.h:171
#6  rd_kafka_topic_destroy_app (app_rkt=0x7f959c002640) at rdkafka_topic.c:84
#7  rd_kafka_topic_destroy (app_rkt=0x7f959c002640) at rdkafka_topic.c:129
#8  0x0000000000440dca in rd_kafka_topic_opaque (app_rkt=<optimized out>) at rdkafka_topic.c:1475
#9  0x000000000040cf7b in RdKafka::MessageImpl::MessageImpl (rkmessage=0x7f959c003c58, this=0x255a250) at rdkafkacpp_int.h:345
#10 RdKafka::KafkaConsumerImpl::consume (this=<optimized out>, timeout_ms=<optimized out>) at KafkaConsumerImpl.cpp:119
#11 0x000000000040a111 in main (argc=<optimized out>, argv=<optimized out>) at rdkafka

[edenhill]

Is this with a completely unmodified v1.4.2 rdkafka_complex_consumer_example ?
I can't reproduce it.

Can you try to reproduce on master?

[dshivashankar1994]

Yes, in master also its reproducible. rdkafka_complex_consumer_example_cpp with no modification.
After subscribing, deleted the topic - "fps.dhamodha.test.hello"

/codemill/dhamodha/librdkafka(master)> /codemill/dhamodha/librdkafka/examples/rdkafka_complex_consumer_example_cpp -b mwkafka-staging-01.nyc.firm.com:9092,mwkafka-staging-02.nyc.firm.com:9092,mwkafka-staging-01.dr.firm.com:9092,mwkafka-staging-02.dr.firm.com:9092,mwkafka-staging-01.tbd.firm.com:909 -g new_cons_gp -X security.protocol=SASL_PLAINTEXT -X sasl.kerberos.service.name=kafka -X sasl.mechanism=GSSAPI -X sasl.kerberos.kinit.cmd='skinit --force' -X client.id=cppcon fps.dhamodha.test.hello 
dhamodha acquired ticket: client dhamodha/hostbased@UNIX.firm.COM, server krbtgt/UNIX.firm.COM@UNIX.firm.COM
% Created consumer cppcon#consumer-1
2020-07-07 19:03:12.476: LOG-3-FAIL: [thrd:sasl_plaintext://mwkafka-staging-01.tbd.firm.com:909/bootstra]: sasl_plaintext://mwkafka-staging-01.tbd.firm.com:909/bootstrap: Connect to ipv4#10.236.30.7:909 failed: Connection refused (after 197ms in state CONNECT)
2020-07-07 19:03:12.476: ERROR (Local: Broker transport failure): sasl_plaintext://mwkafka-staging-01.tbd.firm.com:909/bootstrap: Connect to ipv4#10.236.30.7:909 failed: Connection refused (after 197ms in state CONNECT)
RebalanceCb: Local: Assign partitions: fps.dhamodha.test.hello[0], 
dhamodha acquired ticket: client dhamodha/hostbased@UNIX.firm.COM, server krbtgt/UNIX.firm.COM@UNIX.firm.COM
2020-07-07 19:04:23.228: LOG-5-PARTCNT: [thrd:main]: Topic fps.dhamodha.test.hello partition count changed from 1 to 0
2020-07-07 19:04:23.228: ERROR (Local: Unknown partition): fps.dhamodha.test.hello [0]: desired partition no longer exists (Local: Unknown partition)
2020-07-07 19:04:23.228: ERROR (Broker: Unknown topic or partition): fps.dhamodha.test.hello [0]: topic does not exist (Broker: Unknown topic or partition)
2020-07-07 19:04:23.479: ERROR (Broker: Unknown topic or partition): fps.dhamodha.test.hello [0]: topic does not exist (Broker: Unknown topic or partition)
...
...
2020-07-07 19:08:10.926: ERROR (Broker: Unknown topic or partition): fps.dhamodha.test.hello [0]: topic does not exist (Broker: Unknown topic or partition)
2020-07-07 19:08:11.477: ERROR (Broker: Unknown topic or partition): fps.dhamodha.test.hello [0]: topic does not exist (Broker: Unknown topic or partition)
rdkafka_complex_consumer_example_cpp: rd.h:326: rd_refcnt_sub0: Assertion `!*"refcnt sub-zero"' failed.
Aborted (core dumped)

[edenhill]

Can you reproduce this with ASAN? (build librdkafka with ./dev-conf.sh asan)

[dshivashankar1994]

I don't have libasan installed. I'm restricted from installing it due to regulations.

Are you unable to reproduce the error with the master ?

Please let me know if I can do anything else instead.

[dshivashankar1994]

Hi, Any updates here ?

[dshivashankar1994]

Ping

[edenhill]

Sorry, but I'm not able to reproduce this and I don't have much else to go on if it can't be run in a debugger/valgrind/asan.

[dshivashankar1994]

If this is not reproducible with the same version of the rdkafka example, could it be because of different versions of the kafka broker ?

Also, please note that I have given the output of the debugger (gdb)