Minimum replication factor for QQs

[illotum]

Add ability to set minimum replication factor for quorum queues. Broker picks the maximum of the configured and requested replication factors. Broker does this only on queue creation, and only when replication was requested (including default from policies or config).

Several considerations:

• If implemented via a config option, ensure the purpose is clearly distinct from the existing quorum_cluster_size.
• If implemented via a policy, this has to be added to default operator policies to cover not-yet-existing vhosts where needed.
• Replication decision has to be explicitly logged to avoid confusion.

I'm personally leaning towards doing it with a config.

[michaelklishin]

Going with a new configuration key does not necessarily exclude policy support as the former can be used as a fallback. I'd support both.

[illotum]

Fair enough! If both are implemented, adding the policy to default operator policies becomes optional. Config would fill the same role.

[michaelklishin]

Someone has suggested that we might want to reuse cluster_formation.target_cluster_size_hint as a fallback. I don't have a strong opinion on this but the name already says "cluster_formation.*` and it is not really obvious that it would have an effect on QQs.

[illotum]

A bit wary it will lead to confusion: someone may want to start a cluster with the size_hint of 5 to control definition imports, and will be mighty surprised that three replica queues are disallowed. I'll ponder upon it.

[mkuratczyk]

I like the original idea. I also agree that (re)using target_cluster_size_hint is problematic. There are 5+ node clusters out there with the hint set already (Kubernetes Operator sets it by default these days), so especially in an upgrade scenario, we would suddenly force people to have a QQ member on every node.

[kjnilsson]

I mentioned it in slack but will repeat here. I think we need to design the overall feature of automatic quorum queue membership changes (as I think that is what is ultimately being requested?) first as it can be difficult to evaluate individual configuration settings without at least a draft design.

[kjnilsson]

I need to work on another issue today but will come back to this once finished.

[illotum]

It is both Karl. We do want to be able to maintain the desired replication factor (and balance across AZs eventually), but this issue is just about disallowing replication that may lead to data loss.

[michaelklishin]

Team RabbitMQ decided that this cannot be done in a "one step at a time" manner and needs a single issue that will serve as a design document. I will put together something that would capture our current understanding of the goal of dynamic QQ replica management with the goal of maintaining an operator-set minimum.