`peer:start` is not taking TLS settings into account

[lukebakken]

Describe the bug

Follow-up to: #10135

When building a cluster using inter-node TLS, the peer node won't start, because TLS settings are not being passed to it.

Reproduction steps

git clone git@github.com:lukebakken/docker-rabbitmq-cluster.git
cd docker-rabbitmq-cluster
git checkout tls
git submodule update --init
make RABBITMQ_DOCKER_TAG='pivotalrabbitmq/rabbitmq:f4bf4fe1d1a6071c7ebed751c48e74d1aa9a8420-otp-max-bazel' up

You will see the following:

BOOT FAILED
===========
Exception during startup:

exit:timeout

2023-12-14 19:52:51.427024+00:00 [error] <0.276.0>
2023-12-14 19:52:51.427024+00:00 [error] <0.276.0> BOOT FAILED
2023-12-14 19:52:51.427024+00:00 [error] <0.276.0> ===========
2023-12-14 19:52:51.427024+00:00 [error] <0.276.0> Exception during startup:
2023-12-14 19:52:51.427024+00:00 [error] <0.276.0>
2023-12-14 19:52:51.427024+00:00 [error] <0.276.0> exit:timeout
2023-12-14 19:52:51.427024+00:00 [error] <0.276.0>
2023-12-14 19:52:51.427024+00:00 [error] <0.276.0>     peer:start_it/2, line 646
2023-12-14 19:52:51.427024+00:00 [error] <0.276.0>     rabbit_peer_discovery:query_node_props/1, line 389
2023-12-14 19:52:51.427024+00:00 [error] <0.276.0>     rabbit_peer_discovery:sync_desired_cluster/3, line 188
2023-12-14 19:52:51.427024+00:00 [error] <0.276.0>     rabbit_db:init/0, line 65
2023-12-14 19:52:51.427024+00:00 [error] <0.276.0>     rabbit_boot_steps:-run_step/2-lc$^0/1-0-/2, line 51
2023-12-14 19:52:51.427024+00:00 [error] <0.276.0>     rabbit_boot_steps:run_step/2, line 58
2023-12-14 19:52:51.427024+00:00 [error] <0.276.0>     rabbit_boot_steps:-run_boot_steps/1-lc$^0/1-0-/1, line 22
2023-12-14 19:52:51.427024+00:00 [error] <0.276.0>     rabbit_boot_steps:run_boot_steps/1, line 23
2023-12-14 19:52:51.427024+00:00 [error] <0.276.0>
    peer:start_it/2, line 646
    rabbit_peer_discovery:query_node_props/1, line 389
    rabbit_peer_discovery:sync_desired_cluster/3, line 188
    rabbit_db:init/0, line 65
    rabbit_boot_steps:-run_step/2-lc$^0/1-0-/2, line 51
    rabbit_boot_steps:run_step/2, line 58
    rabbit_boot_steps:-run_boot_steps/1-lc$^0/1-0-/1, line 22
    rabbit_boot_steps:run_boot_steps/1, line 23

Expected behavior

Cluster forms correctly.

Additional context

https://www.erlang.org/doc/apps/ssl/ssl_distribution

TLS related settings like -pa, -ssl_dist_optfile, etc, must be passed to the peer node. Here are some example arguments from a TLS-enabled cluster node:

(rabbit@rmq2.local)2> rp(init:get_arguments()).
[{root,["/usr/local/lib/erlang"]},
 {bindir,["/usr/local/lib/erlang/erts-14.1.1/bin"]},
 {progname,["erl"]},
 {home,["/var/lib/rabbitmq"]},
 {pa,[[]]},
 {noshell,[]},
 {noinput,[]},
 {boot,["start_sasl"]},
 {pa,["/usr/local/lib/erlang/lib/ssl-11.0.3/ebin"]},
 {proto_dist,["inet_tls"]},
 {ssl_dist_optfile,["/etc/rabbitmq/inter_node_tls.config"]},
 {syslog,["logger","[]"]},
 {syslog,["syslog_error_logger","false"]},
 {kernel,["prevent_overlapping_partitions","false"]},
 {'enable-feature',["maybe_expr"]}]
ok

[dumbbell]

Thank you for testing peer discovery further and reporting this second issue! Again, good find!

[lukebakken]

Some notes for getting TLS related options from init:

(rabbit@rmq2.local)1> proplists:to_map(init:get_arguments()).
#{bindir => ["/opt/erlang/lib/erlang/erts-13.2.2.4/bin"],
  boot => ["start_sasl"],
  'enable-feature' => ["maybe_expr"],
  home => ["/var/lib/rabbitmq"],
  kernel => ["prevent_overlapping_partitions","false"],
  noinput => [],noshell => [],
  pa => [[]],
  progname => ["erl"],
  proto_dist => ["inet_tls"],
  root => ["/opt/erlang/lib/erlang"],
  ssl_dist_optfile => ["/etc/rabbitmq/inter_node_tls.config"],
  syslog => ["logger","[]"]}
(rabbit@rmq2.local)2> init:get_argument(proto_dist).
{ok,[["inet_tls"]]}
(rabbit@rmq2.local)3> init:get_argument(pa).
{ok,[[[]],["/opt/erlang/lib/erlang/lib/ssl-10.9.1.2/ebin"]]}
(rabbit@rmq2.local)4> init:get_argument(ssl_dist_optfile).
{ok,[["/etc/rabbitmq/inter_node_tls.config"]]}
(rabbit@rmq2.local)5> init:get_argument(kernel).
{ok,[["prevent_overlapping_partitions","false"]]}
(rabbit@rmq2.local)6>