dead-lock in multi-thread enviroment

[desmondhe]

I create some producers in the same process.
all of my producers are locked , no data can be sent to kafka
and there are many error-callback logs("Local: Message timed out")

gdb info:
........

Thread 25 (Thread 1333913920 (LWP 26214)):
#0 0x00002b1bf8c18c77 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
#1 0x00002b1bfc2f98c6 in rd_kafka_topic_metadata_update (rkb=0x5d6320, tm=0x2aaaab104120) at rdkafka_topic.c:868
#2 0x00002b1bfc2ecd6f in rd_kafka_metadata_handle (rkb=0x5d6320, req_rkt=0x5add10, buf=, size=1333893376) at rdkafka_broker.c:821
#3 0x00002b1bfc2ed270 in rd_kafka_broker_metadata_reply (rkb=0x5d6320, err=0, reply=0x2aaaab102be0, request=0x2aaaab103f60, opaque=0x5d5ed0) at rdkafka_broker.c:864
#4 0x00002b1bfc2ee320 in rd_kafka_recv (rkb=0x5d6320) at rdkafka_broker.c:1101
#5 0x00002b1bfc2ee6d8 in rd_kafka_broker_io_serve (rkb=0x5d6320) at rdkafka_broker.c:2141
#6 0x00002b1bfc2eff08 in rd_kafka_broker_thread_main (arg=) at rdkafka_broker.c:2160
#7 0x00002b1bf8c15193 in start_thread () from /lib64/libpthread.so.0
#8 0x00002b1bf8eedf0d in clone () from /lib64/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 24 (Thread 1342306624 (LWP 26215)):
#0 0x00002b1bf8c18c77 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
#1 0x00002b1bfc2f98c6 in rd_kafka_topic_metadata_update (rkb=0x5d69d0, tm=0x2aaaab10a110) at rdkafka_topic.c:868
#2 0x00002b1bfc2ecd6f in rd_kafka_metadata_handle (rkb=0x5d69d0, req_rkt=0x0, buf=, size=1342281952) at rdkafka_broker.c:821
#3 0x00002b1bfc2ed270 in rd_kafka_broker_metadata_reply (rkb=0x5d69d0, err=0, reply=0x2aaaab1034d0, request=0x2aaaab103ba0, opaque=0x5d5ed0) at rdkafka_broker.c:864
#4 0x00002b1bfc2ee320 in rd_kafka_recv (rkb=0x5d69d0) at rdkafka_broker.c:1101
#5 0x00002b1bfc2ee6d8 in rd_kafka_broker_io_serve (rkb=0x5d69d0) at rdkafka_broker.c:2141
#6 0x00002b1bfc2eff08 in rd_kafka_broker_thread_main (arg=) at rdkafka_broker.c:2160
#7 0x00002b1bf8c15193 in start_thread () from /lib64/libpthread.so.0
#8 0x00002b1bf8eedf0d in clone () from /lib64/libc.so.6
#9 0x0000000000000000 in ?? ()

......

[edenhill]

Which version/sha is this?

[desmondhe]

librdkafka-0.8.3

[edenhill]

Okay, good, this problem was fixed in 0.8.4 (or master).
Can you upgrade and verify it fixes your problem?

[desmondhe]

OK, Let me try

[edenhill]

Any luck?

[desmondhe]

It's normal at this moment
I think it should take more time to be verified, because the problem can't be so easy to reproduce.

[edenhill]

Okay, cool, let me know how it goes.

Back when I fixed it I reproduced this issue by freezing a broker (Ctrl-Z), e.g., letting it time out from its peers.

[desmondhe]

I don't know how to reproduce it
it's usually reproduced about 2 or 3 days

But I will be on vocation from tomorrow till Tuesday, here comes Mid-autumn Day.
I will check it when i'm back

[desmondhe]

It works fine until now.
thanks a lot.

[edenhill]

Awesome!