librdkafka hanging on rd_kafka_destroy()

[hendrik-schumacher]

We have a perl script with xs bindings that runs a kafka producer. This application finishes and calls rd_kafka_destroy().

We see this application hanging forever with the following three threads:

• main thread (with destroy call):
  #0 0x00007f10a792c4db in pthread_join (threadid=139709482862336, thread_return=0x7ffc5a7390e0) at pthread_join.c:92
  Bugfix in example code #1 0x00007f10a6a1a9f1 in thrd_join () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
  Program received signal SIGPIPE, Broken pipe. #2 0x00007f10a69e6eb6 in rd_kafka_destroy () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
  excessive memory allocation == poor performance? #3 0x00007f10a69dfe1e in ?? () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
  ZooKeeper integration #4 0x00007f10a8111d7b in Perl_pp_entersub () from /usr/lib/x86_64-linux-gnu/libperl.so.5.20
  rd_kafka_destroy doesn't seem to close the socket properly #5 0x00007f10a810a5e6 in Perl_runops_standard () from /usr/lib/x86_64-linux-gnu/libperl.so.5.20
  kafka v0.6 support #6 0x00007f10a80934d5 in Perl_call_sv () from /usr/lib/x86_64-linux-gnu/libperl.so.5.20
  make error: rdkafka_example #7 0x00007f10a811ba10 in ?? () from /usr/lib/x86_64-linux-gnu/libperl.so.5.20
  Created a small c++ librdkafka produce wrapper.  #8 0x00007f10a811c430 in Perl_sv_clear () from /usr/lib/x86_64-linux-gnu/libperl.so.5.20
  Fix: Makefile doesn't contained all the headers in HDRS variable #9 0x00007f10a811c7ea in Perl_sv_free2 () from /usr/lib/x86_64-linux-gnu/libperl.so.5.20
  Producer OOM precaution #10 0x00007f10a81411b8 in Perl_leave_scope () from /usr/lib/x86_64-linux-gnu/libperl.so.5.20
  Free payload memory in rd_kafka_example when queue is full #11 0x00007f10a814bb68 in Perl_pp_leave () from /usr/lib/x86_64-linux-gnu/libperl.so.5.20
  Add return code for Kafka::produce and free message payload memory in rd_kafka_example when queue is full #12 0x00007f10a810a5e6 in Perl_runops_standard () from /usr/lib/x86_64-linux-gnu/libperl.so.5.20
  compilation fails if -DWITH_LIBRD present. Fixed. #13 0x00007f10a809b14d in perl_run () from /usr/lib/x86_64-linux-gnu/libperl.so.5.20
  librdkafka 0.8 can not produce messages when the leader of the topic partition failed #14 0x0000000000400e19 in main ()
• second thread also waiting for join:
  #0 0x00007f10a792c4db in pthread_join (threadid=139709466076928, thread_return=0x7f10a618fd60) at pthread_join.c:92
  Bugfix in example code #1 0x00007f10a6a1a9f1 in thrd_join () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
  Program received signal SIGPIPE, Broken pipe. #2 0x00007f10a69e4eab in ?? () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
  excessive memory allocation == poor performance? #3 0x00007f10a69e6576 in ?? () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
  ZooKeeper integration #4 0x00007f10a6a1a5c7 in ?? () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
  rd_kafka_destroy doesn't seem to close the socket properly #5 0x00007f10a792b0a4 in start_thread (arg=0x7f10a6190700) at pthread_create.c:309
  kafka v0.6 support #6 0x00007f10a766087d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
• third thread running some broker io logic:
  #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
  Bugfix in example code #1 0x00007f10a6a1a879 in cnd_timedwait_ms () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
  Program received signal SIGPIPE, Broken pipe. #2 0x00007f10a69fef22 in rd_kafka_q_pop () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
  excessive memory allocation == poor performance? #3 0x00007f10a69ef35b in ?? () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
  ZooKeeper integration #4 0x00007f10a69ef9de in ?? () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
  rd_kafka_destroy doesn't seem to close the socket properly #5 0x00007f10a69efef8 in ?? () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
  kafka v0.6 support #6 0x00007f10a6a1a5c7 in ?? () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
  make error: rdkafka_example #7 0x00007f10a792b0a4 in start_thread (arg=0x7f10a518e700) at pthread_create.c:309
  Created a small c++ librdkafka produce wrapper.  #8 0x00007f10a766087d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

or
#0 0x00007f10a7657d3d in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f10a69fb794 in rd_kafka_transport_poll () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
#2 0x00007f10a69fb7d5 in rd_kafka_transport_io_serve () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
#3 0x00007f10a69ef62a in ?? () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
#4 0x00007f10a69ef9de in ?? () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
#5 0x00007f10a69efef8 in ?? () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
#6 0x00007f10a6a1a5c7 in ?? () from /usr/lib/x86_64-linux-gnu/perl5/5.20/auto/KafkaClient/KafkaClient.so
#7 0x00007f10a792b0a4 in start_thread (arg=0x7f10a518e700) at pthread_create.c:309
#8 0x00007f10a766087d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

We are using librdkafka 0.9.0. It may be possible that we had some broker connectivity issues when the application got stuck. Since the main thread just blocks on join we do not see any way to break out of this state.

[edenhill]

Hi,

Ive fixed a number of lockup-on-termination problems since 0.9.0, could you try the latest master branch to see if the problem remains?

Also make sure all topic_t or message_t objects are properly destroyed prior to calling rd_kafka_destroy().

[hendrik-schumacher]

Thanks for the heads-up. This is only happening for us in certain situations which we could not reproduce outside of production. Is there any release coming up we could deploy? Is master stable enough for selective deployment in production?

[edenhill]

Yes, it is stable enough, a new release off master is imminent. (next week or the week after)

[hendrik-schumacher]

Thanks. We will try out master next week.