NoAck Consumers get socket closed by rabbitmq due to lack of heartbeat

[tom-railpod]

Versions:
rabbitpy 0.27.1
RabbitMQ 3.2.4. on ubuntu 14.04.4

The failure was caused by a connection with only consumers (without an ACK).
This resulted in no back traffic from the consumer to rabbitmq. After the
the timeout period, rabbitmq closed the socket and the consumer died.

Note: As a prior workaround, the connection was created with a timeout of 0
which should have turned off the timeout. rabbitmq appeared not to honor
this as the timeout was listed as 520 seconds.

The fix was to a single rabbitpy file: channel0.py. It was altered to
construct and send a heartbeat message back to rabbitmq on the timeout
interval.

The production GitHub repo of rabbitpy 0.27.1 (https://github.com/gmr/rabbitpy)
was forked to:
(https://github.com/tom-railpod/rabbitpy)
and the changes were made to a single file (channel0.py)

The only enhancement I might suggest is to only send the heartbeat
if there is no data has been recently sent back to rabbitmq. I
was under production delivery pressure so could not adequately
pursue this. On brief examination, I didn't find a "num_of_bytes_sent"
counter and didn't really have a way to test it. There is probably
no performance issue with the code as is.

I've generated a pull request

Funny story, I spent hours finding out this exact same problem in another Python library and came to that conclusion today also. I can confirm that RabbitMQ 3.6.0 & 3.6.1 exhibit this behavior and this sort of patch seems necessary, though I have not use this library.

[eandersson]

I think the solution based on an irc conversation would be to send heartbeats from the client if no data has been sent or received within n/2 of the interval.

I concur. I just interacted with your library in a way that had
unintended consequences. Maybe I was using the wrong connection for the
heartbeat?

On 6/23/2016 11:17 AM, Erik Olof Gunnar Andersson wrote:

I think the solution based on an irc conversation would be to send
heartbeats from the client if no data has been sent or received within
n/2 of the interval.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#98 (comment), or
mute the thread
https://github.com/notifications/unsubscribe/ATKoWZaPaNT3BBA6nOMif-ZWyBup7Kfqks5qOqOhgaJpZM4I7_sQ.

[tom-railpod]

"send heartbeats from the client if no data has been sent or received within n/2 of the interval."

That wouldn't work on the case that failed. There was tons of data coming FROM rabbitmq to the rabbitpy consumer but it was unack'ed. There was just no data going BACK from rabbitpy queue to rabbitmq.

I actually looked (unsuccessfully) for a rabbitmq spec on the heartbeat. If it is as you suggested, then rabbitmq is buggy. I think the right answer is what I mentioned in initial post, i.e. send heartbeat back to rabbitmq if no data has been sent BACK in n/2 of the interval.

"I just interacted with your library in a way that had
unintended consequences" What were the consequences? The only thing I was really worried about was an infinite round-robin echoing of heartbeats as there is code in rabbitpy which echos a heartbeat if if gets one.

Sorry about talking about interacting with a library, thought this was a
different source. I was talking about amqp-storm with the unintended
consequences.

If you note, the RabbitMQ documentation says that the 'official'-ish
.Net and Java clients send heartbeats every interval/2 seconds, which is
another strong argument in favor of that behavior.
https://www.rabbitmq.com/heartbeats.html

On 6/23/2016 1:35 PM, tom campbell wrote:

"send heartbeats from the client if no data has been sent or received
within n/2 of the interval."

That wouldn't work on the case that failed. There was tons of data
coming FROM rabbitmq to the rabbitpy consumer but it was unack'ed.
There was just no data going BACK from rabbitpy queue to rabbitmq.

I actually looked (unsuccessfully) for a rabbitmq spec on the
heartbeat. If it is as you suggested, then rabbitmq is buggy. I think
the right answer is what I mentioned in initial post, i.e. send
heartbeat back to rabbitmq if no data has been sent BACK in n/2 of the
interval.

"I just interacted with your library in a way that had
unintended consequences" What were the consequences? The only thing I
was really worried about was an infinite round-robin echoing of
heartbeats as there is code in rabbitpy which echos a heartbeat if if
gets one.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#98 (comment), or
mute the thread
https://github.com/notifications/unsubscribe/ATKoWQQIkTlPSAxHX4s3aNtUxj8xCBAVks5qOsPfgaJpZM4I7_sQ.

[gmr]

This is addressed in 1.0