Can you explain a bit about the 'key' passed by the producer?

[winbatch]

I know it can be used by the partitioner, but I'm also wondering what other value it has - I read that it is also passed to the consumer. Could this be considered like a 'header field'? (thinking like a JMS Header field). Is there only a single key that can be passed? If so, I guess I would have to make it somewhat of a compound/concatenated/delimited one if I want to have a bunch of properties passed?

[edenhill]

The main use of the key is for partitioning: it is used in combination with a specific partitioner to make sure that messages with a certain key goes to the same partition.
In this case the partitioner can be considered a hashing function, it takes a possibly per-message unique key and boils it down to a specific partition.

For example, say that we are making a user authentication monitor and for each login attempt a message is logged through kafka. The key is set to the username.
If we write a partitioner that hashes the username and applies modulu to the hashed value, we know that each message produced for a specific user will go to the same partition.

On the other hand if all we need is load balancing - spread messages equally over all partitioners - then the key is
usually not of interest and the default random partitioner can be used instead.

You could use it as a header, the key is just a binary blob, the broker does not act on the key in any way, it is simply attached to the message and provided with the message to consumers.

[winbatch]

sounds good, thanks. The reason I want to use it as a header is that I need to add some metadata about what it is I'm sending as the message, and I don't want to send that metadata as part of the actual message payload (for a couple of reasons. 1 - I want the information to be optional and the consumer to ignore it if he doesn't care. 2 - I don't want to have to do any 'escaping' of the payload contents due to the nature of the 'envelope' I would have had to create inside the payload. For example, if my envelope was in JSON, I would then have to ensure that my payload was escaped to avoid JSON parson problems, etc.).