Crash on exit

[winbatch]

I'm trying to exit cleanly, and periodically a get a core dump that looks like this:
#0 0x00007f4f51cea8d8 in _fini () from /lib64/libresolv.so.2
#1 0x00000039fa60eb6c in _dl_fini () from /lib64/ld-linux-x86-64.so.2
#2 0x00000039fae35d92 in exit () from /lib64/libc.so.6

(more stuff below)

This is my shutdown code. Is it valid? should I be doing something else to wait for kafka to exit cleanly? (note PluginReturn is just an enum)

PluginReturn LaraTargetShutdownFunc()
{
CommonConfig::rLog( LWARNING, "[%s] Shutting down kafka components\n", cfg->section.c_str());
rd_kafka_topic_destroy (rkt);
rd_kafka_destroy (rk);
int ret=0;
while ( 1 )
{
CommonConfig::rLog( LWARNING, "[%s] Waiting for kafka components to shutdown...\n", cfg->section.c_str());
ret = rd_kafka_wait_destroyed (1000);
if ( ret == 0 )
break;
}
CommonConfig::rLog( LWARNING, "[%s] Shutdown of kafka components complete\n", cfg->section.c_str());
return PLUGIN_SUCCESS;
}

[winbatch]

And there are cases where it is stuck over and over in the 'Waiting' loop....

[edenhill]

I fixed a topic refcnt problem on a private branch a couple of days ago. This is probably what you are seeing.
Will push it soon.

[winbatch]

That's be great. Is my logic ok?

On Wednesday, February 5, 2014, Magnus Edenhill notifications@github.com
wrote:

I fixed a topic refcnt problem on a private branch a couple of days ago.
This is probably what you are seeing.
Will push it soon.

Reply to this email directly or view it on GitHubhttps://github.com//issues/72#issuecomment-34286662
.

[edenhill]

Almost!

For a producer you will also want to check that all messages have been sent (unless you control this yourself by the dr_cb):

 /* Shutdown */

 /* Wait for messages to be sent */
 while (rd_kafka_outq_len(rk) > 0)
   rd_kafka_poll(rk, 1000);

rd_kafka_topic_destroy(rkt);
rd_kafka_destroy(rkt);

/* And if you want a clean (all memory freed) shutdown, also wait for the
  * background destroy to finish */

rd_kafka_wait_destroyed(5000);

[winbatch]

I'm already doing the dr_db bit. My question was more around the loop. If I want to guarantee that everything shuts down cleanly, should I do a loop or just put a very high number in the wait_destroyed() function?

[edenhill]

The only reason for calling wait_destroyed() repeatedly is if your application wants to do something, otherwise just put a larger number for the wait_destroyed().
Having said that, the destroy should be fairly instant, the only reason for a slow shutdown I can think of is if a broker is completely unreachable so the connect(2) call has to time out, but that should be fairly rare.

[winbatch]

There were cases where it was in a constant loop, even at 1000 ms
intervals. Might be (hopefully) related to the refcount thing?

On Thu, Feb 6, 2014 at 6:50 AM, Magnus Edenhill notifications@github.comwrote:

The only reason for calling wait_destroyed() repeatedly is if your
application wants to do something, otherwise just put a larger number for
the wait_destroyed().
Having said that, the destroy should be fairly instant, the only reason
for a slow shutdown I can think of is if a broker is completely unreachable
so the connect(2) call has to time out, but that should be fairly rare.

Reply to this email directly or view it on GitHubhttps://github.com//issues/72#issuecomment-34316003
.

[edenhill]

Yep, as the topic refcnt was not properly decreased in all situations it
would lead to rdkafka never terminating completely.
Should be fixed.

2014-02-06 winbatch notifications@github.com:

There were cases where it was in a constant loop, even at 1000 ms
intervals. Might be (hopefully) related to the refcount thing?

On Thu, Feb 6, 2014 at 6:50 AM, Magnus Edenhill <notifications@github.com

wrote:

The only reason for calling wait_destroyed() repeatedly is if your
application wants to do something, otherwise just put a larger number for
the wait_destroyed().
Having said that, the destroy should be fairly instant, the only reason
for a slow shutdown I can think of is if a broker is completely
unreachable
so the connect(2) call has to time out, but that should be fairly rare.

Reply to this email directly or view it on GitHub<
https://github.com/edenhill/librdkafka/issues/72#issuecomment-34316003>

.

Reply to this email directly or view it on GitHubhttps://github.com//issues/72#issuecomment-34316167
.

[winbatch]

Looks like there are still cases where it won't shut down regardless of wait time

[winbatch]

And cases where it still crashes. replicator: rdkafka_topic.c:323: rd_kafka_topic_destroy0: Assertion `rkt->rkt_refcnt == 0' failed.

Program terminated with signal 11, Segmentation fault.

#0 0x00000039fae7788a in _int_free () from /lib64/libc.so.6

Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.47.el6_2.9.x86_64 libgcc-4.4.6-3.el6.x86_64 libstdc++-4.4.6-3.el6.x86_64 zlib-1.2.3-27.el6.x86_64

(gdb) where

#0 0x00000039fae7788a in _int_free () from /lib64/libc.so.6

#1 0x00007fa3974813a6 in rd_kafka_destroy0 (rk=0x39fb18be80) at rdkafka.c:570

#2 0x00007fa39748beff in rd_kafka_topic_destroy0 (rkt=0x26e3ba0) at rdkafka_topic.c:326

#3 0x00007fa39748c0ee in ?? () at rdkafka_msg.h:100 from ../lib/liblarakafka.so

#4 0x00007fa388004000 in ?? ()

#5 0x00007fa3974884c5 in rd_kafka_broker_fail (rkb=0x7fa388002ec0, err=RD_KAFKA_RESP_ERR__DESTROY, fmt=0x0) at rdkafka_broker.c:386

#6 0x00007fa39748a79b in rd_kafka_broker_thread_main (arg=0x7fa388002ec0) at rdkafka_broker.c:3278

#7 0x00000039fb6077f1 in start_thread () from /lib64/libpthread.so.0

#8 0x00000039faee5ccd in clone () from /lib64/libc.so.6

[edenhill]

Can you call rd_kafka_dump(rk); prior to destroying your topic(s) and the main handle=

And can you provide an example program that reproduces this? (e.g, patch on rdkafka_example)

[winbatch]

So this is interesting. There are certain processes when killed, they
create the dump file (I open the FILE handle before giving it to the dump
program), but the file is empty. It's stuck in some sort of tight loop:

strace shows this:

futex(0xcdc2cc, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 30599,
{1392089159, 435281000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)

futex(0xcdc2a0, FUTEX_WAKE_PRIVATE, 1) = 0

futex(0xcdc2cc, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 30601,
{1392089159, 445411000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)

futex(0xcdc2a0, FUTEX_WAKE_PRIVATE, 1) = 0

gcore reveals:
Core was generated by `./replicator -c ../devconfig/superkafka.ini'.
#0 0x000000345700dff4 in __lll_lock_wait () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.47.el6_2.9.x86_64 libgcc-4.4.6-3.el6.x86_64 libstdc++-4.4.6-3.el6.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) where
#0 0x000000345700dff4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003457009328 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00000034570091f7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f181cfe533a in rd_kafka_topic_scan_all (rk=0xcdc2a0, now=16795998647784) at rdkafka_topic.c:931
#4 0x00007f181cfdb116 in rd_kafka_topic_scan_tmr_cb (rk=0xcdc2a0, arg=) at rdkafka.c:816
#5 0x00007f181cfe7ab5 in rd_kafka_timers_run (rk=0xcdc2a0, timeout=) at rdkafka_timer.c:155
#6 0x00007f181cfda8bd in rd_kafka_thread_main (arg=0xcdc2a0) at rdkafka.c:840
#7 0x00000034570077f1 in start_thread () from /lib64/libpthread.so.0
#8 0x00000034568e5ccd in clone () from /lib64/libc.so.6

On Mon, Feb 10, 2014 at 9:01 PM, Magnus Edenhill
notifications@github.comwrote:

Can you call rd_kafka_dump(rk); prior to destroying your topic(s) and the
main handle=

And can you provide an example program that reproduces this? (e.g, patch
on rdkafka_example)

Reply to this email directly or view it on GitHubhttps://github.com//issues/72#issuecomment-34719780
.