Snappy compression

[winbatch]

Magnus,

I've switched from gzip compression in librdkafka to snappy. Performance has improved dramatically. However, I'm finding that the java Simple Consumer api is unable to consume the messages (it's as if they are not there - that it has reached the end of the partition), while librdkkafka as well as the java High Level consumer are able to consume them fine. Have you ever seen anything like this? Note that the simple consumer had no problem consuming gzip messages.

[edenhill]

That is strange indeed and Ive never seen such things.
Does it tell you anything if you increase the log4j tracing?

[winbatch]

I'm actually emailing with Neha at linkedin. Will see what she says

On Wednesday, February 26, 2014, Magnus Edenhill notifications@github.com
wrote:

That is strange indeed and Ive never seen such things.
Does it tell you anything if you increase the log4j tracing?

Reply to this email directly or view it on GitHubhttps://github.com//issues/89#issuecomment-36204343
.

[edenhill]

For reference, as stated on the mailing list:
The consumer side fetch size must match the producer side max message size.

[winbatch]

There may still be something for you to look at. I have to get you a reproducible case - but there were at least 2 occassions where I did not get an error callback in librdkafka when a message being sent via snappy was too large for the broker. The broker had a stack trace indicating as such. I haven't been able to reproduce it with the rdkafka_* programs yet. I have a feeling there is some sort of 'sweet spot' where the original did not compress well enough and with protocol headers, etc it was too big for kafka. some sort of borderline case.

[edenhill]

Skip compression and specify -X message.max.bytes=N -s N directly to provoke the error you think you might have seen. Should be easier, e.g.:

for N in $(seq 1 1000000 1000) ; do rdkafka_performance -X message.max..=$N -s $N -c 1 ... || break ; done

[winbatch]

Why do you want me to skip compression- my belief is that this happens only
with snappy.

On Thu, Feb 27, 2014 at 8:00 AM, Magnus Edenhill
notifications@github.comwrote:

Skip compression and specify -X message.max.bytes=N -s N directly to
provoke the error you think you might have seen. Should be easier, e.g.:

for N in $(seq 1 1000000 1000) ; do rdkafka_performance -X message.max..=$N -s $N -c 1 ... || break ; done

Reply to this email directly or view it on GitHubhttps://github.com//issues/89#issuecomment-36239119
.

[edenhill]

If its a broker-or-rdkafka-does-not-error-correctly-when-message-is-just-too-large it shouldnt matter if compression is enabled or not. But if you think it is directly snappy related, then disregard my comment.

[winbatch]

what I mean is I believe you check some local limit first and that maybe
with snappy that limit is not being checked?

On Thu, Feb 27, 2014 at 8:04 AM, Magnus Edenhill
notifications@github.comwrote:

If its a
broker-or-rdkafka-does-not-error-correctly-when-message-is-just-too-large
it shouldnt matter if compression is enabled or not. But if you think it is
directly snappy related, then disregard my comment.

Reply to this email directly or view it on GitHubhttps://github.com//issues/89#issuecomment-36239379
.

[winbatch]

In my case I only send 1 msg at a time. Does that change anything in this
regard/make it more brittle?

On Thursday, February 27, 2014, Magnus Edenhill notifications@github.com
wrote:

It is actually quite cowardly:
it first accumulates as many messages into a messageset that will fit
message.max.bytes,
then it compresses that messageset.
So there is a risk that the compressed messageset is actually larger than
the uncompressed message set, but that risk should be extremely thin unless
your data is severly uncompressable.

Reply to this email directly or view it on GitHubhttps://github.com//issues/89#issuecomment-36242733
.

[edenhill]

Could you enable debug=msg and look for this line:
produce messageset with %i messages (NNN)
where %i should be 1 and NNN the compressed message size.
if NNN exceeds message.max.bytes, then thats the problem.

[edenhill]

Any news on this?