`pre_init failed in system quorum_queues` when upgrading from 3.13.4 to 3.13.5

[headyj]

Describe the bug

when changing from rabbitmq:3.13.5-management to rabbitmq:3.13.5-management, the following error appears:

rabbitmq 2024-07-23 10:10:48.033045+00:00 [error] <0.268.0> pre_init failed in system quorum_queues for UId ORCHESLVOCD9VDIAR9 with name *****_898e605a-33e3-4129-9906-f6f46a65a695 This error may need manual intervention
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>   crasher:
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>     initial call: ra_log_pre_init:init/1
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>     pid: <0.268.0>
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>     registered_name: ]
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>     exception exit: {error,{badmatch,false}}
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>       in function  gen_server:init_it/6 (gen_server.erl, line 961)
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>     ancestors: [ra_log_sup,<0.261.0>,ra_systems_sup,ra_sup,<0.208.0>]
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>     message_queue_len: 0
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>     messages: ]
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>     links: [<0.267.0>]
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>     dictionary: ]
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>     trap_exit: false
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>     status: running
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>     heap_size: 1598
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>     stack_size: 28
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>     reductions: 31349
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0>   neighbours:
rabbitmq 2024-07-23 10:10:48.033175+00:00 [error] <0.268.0> 
rabbitmq 2024-07-23 10:10:48.042356+00:00 [error] <0.267.0>     supervisor: {local,ra_log_sup}
rabbitmq 2024-07-23 10:10:48.042356+00:00 [error] <0.267.0>     errorContext: start_error
rabbitmq 2024-07-23 10:10:48.042356+00:00 [error] <0.267.0>     reason: {error,{badmatch,false}}
rabbitmq 2024-07-23 10:10:48.042356+00:00 [error] <0.267.0>     offender: [{pid,undefined},
rabbitmq 2024-07-23 10:10:48.042356+00:00 [error] <0.267.0>                {id,ra_log_pre_init},
rabbitmq 2024-07-23 10:10:48.042356+00:00 [error] <0.267.0>                {mfargs,{ra_log_pre_init,start_link,[quorum_queues]}},
rabbitmq 2024-07-23 10:10:48.042356+00:00 [error] <0.267.0>                {restart_type,permanent},
rabbitmq 2024-07-23 10:10:48.042356+00:00 [error] <0.267.0>                {significant,false},
rabbitmq 2024-07-23 10:10:48.042356+00:00 [error] <0.267.0>                {shutdown,5000},
rabbitmq 2024-07-23 10:10:48.042356+00:00 [error] <0.267.0>                {child_type,worker}]
rabbitmq 2024-07-23 10:10:48.042356+00:00 [error] <0.267.0> 
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>     supervisor: {<0.261.0>,ra_system_sup}
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>     errorContext: start_error
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>     reason: {shutdown,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                 {failed_to_start_child,ra_log_pre_init,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                     {error,{badmatch,false}}}}
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>     offender: [{pid,undefined},
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                {id,ra_log_sup},
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                {mfargs,{ra_log_sup,start_link,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                    [#{message_queue_data => off_heap,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       name => quorum_queues,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       names =>
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                           #{directory => ra_directory,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                             closed_mem_tbls =>
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                                 ra_log_closed_mem_tables,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                             directory_rev =>
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                                 ra_directory_reverse,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                             log_ets => ra_log_ets,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                             log_meta => ra_log_meta,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                             log_sup => ra_log_sup,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                             open_mem_tbls =>
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                                 ra_log_open_mem_tables,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                             segment_writer =>
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                                 ra_log_segment_writer,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                             server_sup => ra_server_sup_sup,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                             wal => ra_log_wal,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                             wal_sup => ra_log_wal_sup},
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       data_dir =>
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                           "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-server-2.rabbitmq-nodes.rabbitmq/quorum/rabbit@rabbitmq-server-2.rabbitmq-nodes.rabbitmq",
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       wal_max_size_bytes => 64000000,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       wal_max_batch_size => 4096,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       compress_mem_tables => false,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       default_max_append_entries_rpc_batch_size =>
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                           16,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       segment_compute_checksums => true,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       wal_compute_checksums => true,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       wal_data_dir =>
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                           "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-server-2.rabbitmq-nodes.rabbitmq/quorum/rabbit@rabbitmq-server-2.rabbitmq-nodes.rabbitmq",
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       segment_max_entries => 4096,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       segment_max_pending => 1024,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       wal_max_entries => undefined,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       wal_write_strategy => default,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       wal_sync_method => datasync,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       wal_garbage_collect => false,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       wal_pre_allocate => false,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       wal_min_bin_vheap_size => 46422,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       wal_min_heap_size => 233,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       server_min_heap_size => 233,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       server_min_bin_vheap_size => 46422,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       default_max_pipeline_count => 4096,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       snapshot_chunk_size => 1000000,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       receive_snapshot_timeout => 30000,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       low_priority_commands_flush_size => 16,
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                       low_priority_commands_in_memory_size =>
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                                           16}]}},
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                {restart_type,permanent},
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                {significant,false},
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                {shutdown,infinity},
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0>                {child_type,supervisor}]
rabbitmq 2024-07-23 10:10:48.042631+00:00 [error] <0.261.0> 
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0> Failed to start Ra system "quorum_queues": {error,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                             {{shutdown,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                               {failed_to_start_child,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                ra_log_sup,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                {shutdown,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                 {failed_to_start_child,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                  ra_log_pre_init,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                  {error,{badmatch,false}}}}}},
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                              {child,undefined,quorum_queues,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                               {ra_system_sup,start_link,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                [#{message_queue_data =>
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                    off_heap,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                   name => quorum_queues,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                   names =>
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                    #{directory => ra_directory,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                      closed_mem_tbls =>
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                       ra_log_closed_mem_tables,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                      directory_rev =>
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                       ra_directory_reverse,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                      log_ets => ra_log_ets,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                      log_meta => ra_log_meta,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                      log_sup => ra_log_sup,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                      open_mem_tbls =>
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                       ra_log_open_mem_tables,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                      segment_writer =>
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                       ra_log_segment_writer,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                      server_sup =>
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                       ra_server_sup_sup,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                      wal => ra_log_wal,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                      wal_sup =>
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                       ra_log_wal_sup},
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                   data_dir =>
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                    "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-server-2.rabbitmq-nodes.rabbitmq/quorum/rabbit@rabbitmq-server-2.rabbitmq-nodes.rabbitmq",
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                   wal_max_size_bytes =>
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                    64000000,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                   wal_max_batch_size => 4096,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                   compress_mem_tables => false,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                   default_max_append_entries_rpc_batch_size =>
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                    16,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                   segment_compute_checksums =>
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                    true,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                   wal_compute_checksums =>
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                    true,
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                   wal_data_dir =>
rabbitmq 2024-07-23 10:10:48.043801+00:00 [error] <0.254.0>                                                    "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-server-2.rabbitmq-nodes.rabbitmq/quorum/rabbit@rabbitmq-server-2.rabbitmq-nodes.rabbitmq",

Reproduction steps

1. Upgrade from 3.13.4 to 3.13.5

Expected behavior

Pod should start normally

Additional context

• Using RabbitMQ Cluster Operator (v4.3.14 - https://artifacthub.io/packages/helm/bitnami/rabbitmq-cluster-operator) on Kubernetes v1.29 (EKS)
• All stable features flags are enabled
• Enabled plugins:
  • rabbitmq_web_mqtt
  • rabbitmq_auth_mechanism_ssl
  • rabbitmq_auth_backend_http
  • rabbitmq_prometheus
• Additional config:

vm_memory_high_watermark.relative = 0.7
auth_backends.1           = internal
auth_backends.2           = http
auth_http.http_method     = post
auth_http.user_path       = http://rabbitmq-auth-proxy-server.rabbitmq-auth-server.svc.cluster.local/auth/user
auth_http.vhost_path      = http://rabbitmq-auth-proxy-server.rabbitmq-auth-server.svc.cluster.local/auth/vhost
auth_http.resource_path   = http://rabbitmq-auth-proxy-server.rabbitmq-auth-server.svc.cluster.local/auth/resource
auth_http.topic_path      = http://rabbitmq-auth-proxy-server.rabbitmq-auth-server.svc.cluster.local/auth/topic
raft.wal_max_size_bytes   = 64000000

[kjnilsson]

Do you have a directory named ORCHESLVOCD9VDIAR9 in your /mnesia/NODE/quorum/NODE/ directory? And if so does it in turn have a snapshots directory?

[headyj]

Yes:

/ORCHESLVOCD9VDIAR9/snapshots$ ls -R
.:
0000000000000001_0000000000000015

./0000000000000001_0000000000000015:
snapshot.dat

[kjnilsson]

can you share that directory with me (ORCHESLVOCD9VDIAR9)?

[kjnilsson]

Does /ORCHESLVOCD9VDIAR9/checkpoints exist?

[headyj]

can you share that directory with me (ORCHESLVOCD9VDIAR9)?

Yes, I sent it to you by mail

Does /ORCHESLVOCD9VDIAR9/checkpoints exist?

No, but we have a lot of other directories with a checkpoints directory in it

[kjnilsson]

If the checkpoints directory does not exist this means ORCHESLVOCD9VDIAR9 must refer to a queue member that is no longer used, perhaps there was a failed deletion in the past or something.

You can try to manually create the checkpoints directory in ORCHESLVOCD9VDIAR9 (and check it exists in all folders in /mnesia/NODE/quorum/NODE/

[headyj]

It doesn't seems to work. Basically I did this command on all my nodes:

cd /var/lib/rabbitmq/mnesia/<node>/quorum/<node>
for d in */ ; do
    mkdir -p $d/checkpoints
done

But when I try to upgrade to v3.13.5, I have the same errors but with a different directory name, which actually doesn't exist at all: ORCHESTFNJNV86GHV6

[kjnilsson]

have you ever manually deleted any of these directories in the past?

[headyj]

No, but for the context, we have approximately 900 queues, most of which are quorum queues. The number of queues fluctuates with the number of connected users on our platform, as each connection has its own queue until it disconnects. Additionally, we have approximately 80 namespaces. Some connections are over AMQP, and some are over MQTT (using the MQTT plugin). We also use the auth backend HTTP plugin for authenticating MQTT connections.

Maybe it can help...

[kjnilsson]

I would recommend skipping this version for now. We're working on some fixes.

[headyj]

OK, do you think we should go back to v3.13.3 ? I'm just asking because of the release notes of 3.13.5:

This upgrade is highly recommended to all users currently on earlier 3.13.x series and
in particular 3.13.4.

[michaelklishin]

#11794

[michaelklishin]

@headyj or 3.13.2. 3.13.5 addresses something related but a different issue.

[michaelklishin]

Should be addressed by #11794