Deadlock when calling lasp:query/1

[NAR]

I'm working on an application that (in my test) runs in a Kubernetes stateful set on 4 pods. There's one Erlang node in each pod. The nodes (in this test) have many processes, but only one process (a gen_server) that reads and writes some data to/from lasp. The reason why these processes store this data in lasp is that these processes might get migrated to a different node and if they are started on the new node, they need the data from the old node. There's no concurrent access to these variables even from within a single node (before a previous refactoring step they we in the state of the gen_server). These processes are also registered using lasp_pg (I don't know if it matters). Each process has 3 variables (a gcounter, an awmap and an awset_ps). The ID of the gcounter variable on node0 is {key0, latest_committed}, on node1 it's {key1, latest_committed}, etc. These are separate variables with possibly separate values, I'm just adding it here because they show up the traces below.

For some reason the system gets stuck on a lasp:query call for the {key0, latest_committed} variable on node0. I made backtraces on the processes in the system and it looks like:

1, we're making a lasp:query call that will eventually make a gen_server:call towards the lasp_distribution_backend process.
2, lasp_distribution_backend process makes a gen_server:call towards lasp_storage_backend.
3, lasp_storage_backend process makes a gen_server:call towards lasp_ets_storage_backend.
4, lasp_ets_storage_backend process makes a gen_server:call towards partisan_pluggable_peer_service_manager.
5, partisan_pluggable_peer_service_manager process makes some kind of call towards partisan_peer_service_events. I haven’t found actually this call in the code yet, the backtrace shows gen_event:rpc/2.
6, partisan_peer_service_events makes a gen_server:call towards lasp_distribution_backend - and we have a cycle, back to step 2.

The backtrace from the gen_server making the lasp:query call:

Program counter: 0x00007f695609fd48 (gen:do_call/4 + 336)
0x00007f6950e3b300 []
0x00007f6950e3b308 infinity
0x00007f6950e3b310 []
0x00007f6950e3b318 #Ref<0.1669894692.3688366084.43478>
0x00007f6950e3b320 <0.1482.0>

0x00007f6950e3b328 Return addr 0x00007f6956090498 (gen_server:call/3 + 128)
y(0)     infinity
y(1)     {query,{{key0,latest_committed},gcounter}}
y(2)     lasp_distribution_backend
y(3)     Catch 0x00007f6956090498 (gen_server:call/3 + 128)

Process 1482 is lasp_distribution_backend. Backtrace:

Program counter: 0x00007f3153b3aa78 (gen:do_call/4 + 336)
0x00007f3150ff10e8 []
0x00007f3150ff10f0 infinity
0x00007f3150ff10f8 []
0x00007f3150ff1100 #Ref<0.2459672142.441188354.235755>
0x00007f3150ff1108 <0.1484.0>

0x00007f3150ff1110 Return addr 0x00007f31536cc608 (gen_server:call/3 + 128)
y(0)     infinity
y(1)     {get,{<<"_membership">>,lwwregister}}
y(2)     <0.1484.0>
y(3)     Catch 0x00007f31536cc608 (gen_server:call/3 + 128)

0x00007f3150ff1138 Return addr 0x00007f3151a2aad8 (lasp_core:declare/5 + 136)
y(0)     []
y(1)     []
y(2)     <0.1484.0>
y(3)     []
y(4)     #Fun<lasp_core.22.57498702>
y(5)     lwwregister
y(6)     {<<"_membership">>,lwwregister}

0x00007f3150ff1178 Return addr 0x00007f31512b1ae8 (lasp_distribution_backend:handle_call/3 + 8704)
y(0)     {state,<0.1484.0>,[],<<118,19,156,189,20,175,97,33,29,169,48,224,253,193,67,22,33,202,214,121>>}

0x00007f3150ff1188 Return addr 0x00007f31536d0cb0 (gen_server:try_handle_call/4 + 136)
y(0)     []
y(1)     []
y(2)     Catch 0x00007f31536d0cf8 (gen_server:try_handle_call/4 + 208)

0x00007f3150ff11a8 Return addr 0x00007f31536d10e0 (gen_server:handle_msg/6 + 168)
y(0)     []
y(1)     []
y(2)     []
y(3)     {declare_dynamic,{<<"_membership">>,lwwregister},lwwregister}
y(4)     {<0.1410.0>,#Ref<0.2459672142.441188356.233525>}
y(5)     infinity
y(6)     lasp_distribution_backend
y(7)     {state,<0.1484.0>,[],<<118,19,156,189,20,175,97,33,29,169,48,224,253,193,67,22,33,202,214,121>>}
y(8)     lasp_distribution_backend
y(9)     <0.1436.0>

Process 1484 is lasp_storage_backend
backtrace:

Program counter: 0x00007f3153b3aa78 (gen:do_call/4 + 336)
0x00007f3150fbf480 []
0x00007f3150fbf488 infinity
0x00007f3150fbf490 []
0x00007f3150fbf498 #Ref<0.2459672142.441188353.235451>
0x00007f3150fbf4a0 <0.1485.0>

0x00007f3150fbf4a8 Return addr 0x00007f31536cc608 (gen_server:call/3 + 128)
y(0)     infinity
y(1)     {fold,#Fun<lasp_state_based_synchronization_backend.12.63600656>,[]}
y(2)     <0.1485.0>
y(3)     Catch 0x00007f31536cc608 (gen_server:call/3 + 128)

0x00007f3150fbf4d0 Return addr 0x00007f31533d4100 (lasp_storage_backend:handle_call/3 + 720)
y(0)     {state,lasp_ets_storage_backend,<0.1485.0>}

0x00007f3150fbf4e0 Return addr 0x00007f31536d0cb0 (gen_server:try_handle_call/4 + 136)
y(0)     []
y(1)     []
y(2)     Catch 0x00007f31536d0cf8 (gen_server:try_handle_call/4 + 208)

0x00007f3150fbf500 Return addr 0x00007f31536d10e0 (gen_server:handle_msg/6 + 168)
y(0)     []
y(1)     []
y(2)     []
y(3)     {fold,#Fun<lasp_state_based_synchronization_backend.12.63600656>,[]}
y(4)     {<0.1486.0>,#Ref<0.2459672142.441188356.233514>}
y(5)     infinity
y(6)     lasp_storage_backend
y(7)     {state,lasp_ets_storage_backend,<0.1485.0>}
y(8)     lasp_storage_backend
y(9)     <0.1482.0>

Process 1485 is lasp_ets_storage_backend
backtrace:

Program counter: 0x00007f3153b3aa78 (gen:do_call/4 + 336)
0x00007f314eae5088 []
0x00007f314eae5090 infinity
0x00007f314eae5098 []
0x00007f314eae50a0 #Ref<0.2459672142.441188353.235456>
0x00007f314eae50a8 <0.2806.0>

0x00007f314eae50b0 Return addr 0x00007f31536cc608 (gen_server:call/3 + 128)
y(0)     infinity
y(1)     {forward_message,'service_name@host0.network.default.svc.cluster.local',undefined,undefined,undefined,lasp_state_based_synchronization_backend,{'$gen_cast',{state_send,'service_name@host2.default.svc.cluster.local',{{{key1,latest_committed},gcounter},gcounter,[{clock,[{<<118,19,156,189,20,175,97,33,29,169,48,224,253,193,67,22,33,202,214,121>>,23}]}],{state_gcounter,[{[<39828.1685.0>],1},{[<0.1685.0>],1},{[<39909.1687.0>],1}]}},false}},[]}
y(2)     partisan_pluggable_peer_service_manager
y(3)     Catch 0x00007f31536cc608 (gen_server:call/3 + 128)

0x00007f314eae50d8 Return addr 0x00007f31519bab28 (partisan_pluggable_peer_service_manager:cast_message/4 + 96)

0x00007f314eae50e0 Return addr 0x00007f31533e1358 (lasp_synchronization_backend:send/3 + 240)

0x00007f314eae50e8 Return addr 0x00007f31512dd160 (lasp_state_based_synchronization_backend:'-init_state_sync/4-fun-0-'/6 + 736)
y(0)     [{{key0,latest_committed},gcounter}]
y(1)     {{key1,latest_committed},gcounter}

0x00007f314eae5100 Return addr 0x00007f3158e522c8 (lists:foldl/3 + 112)
y(0)     []
y(1)     #Fun<lasp_state_based_synchronization_backend.12.63600656>

0x00007f314eae5118 Return addr 0x00007f3153b2c568 (ets:do_foldl/4 + 208)
y(0)     'service_name@host2.network.default.svc.cluster.local'
y(1)     {{key1,latest_committed},gcounter}
y(2)     []
y(3)     #Fun<lasp_state_based_synchronization_backend.12.63600656>

0x00007f314eae5140 Return addr 0x00007f3153b2c3a0 (ets:foldl/3 + 240)
y(0)     'service_name@host2.network.default.svc.cluster.local'
y(1)     []
y(2)     []
y(3)     Catch 0x00007f3153b2c410 (ets:foldl/3 + 352)

0x00007f314eae5168 Return addr 0x00007f3151a517e8 (lasp_ets_storage_backend:handle_call/3 + 968)
y(0)     {state,'service_name@host2.network.default.svc.cluster.local'}

0x00007f314eae5178 Return addr 0x00007f31536d0cb0 (gen_server:try_handle_call/4 + 136)
y(0)     []
y(1)     []
y(2)     Catch 0x00007f31536d0cf8 (gen_server:try_handle_call/4 + 208)

0x00007f314eae5198 Return addr 0x00007f31536d10e0 (gen_server:handle_msg/6 + 168)
y(0)     []
y(1)     []
y(2)     []
y(3)     {fold,#Fun<lasp_state_based_synchronization_backend.12.63600656>,[]}
y(4)     {<0.1484.0>,#Ref<0.2459672142.441188353.235451>}
y(5)     infinity
y(6)     lasp_ets_storage_backend
y(7)     {state,'service_name@host2.network.default.svc.cluster.local'}
y(8)     lasp_ets_storage_backend
y(9)     <0.1484.0>

Process 2806 is partisan_pluggable_peer_service_manager
backtrace:

Program counter: 0x00007f3153b3aa78 (gen:do_call/4 + 336)
0x00007f3152378128 []
0x00007f3152378130 infinity
0x00007f3152378138 []
0x00007f3152378140 #Ref<0.2459672142.441188356.233524>
0x00007f3152378148 <0.1410.0>
0x00007f3152378150 Return addr 0x00007f3153b3ee30 (gen_event:rpc/2 + 112)

0x00007f3152378158 Return addr 0x00007f31519c1888 (partisan_pluggable_peer_service_manager:handle_info/2 + 1712)
y(0)     []
y(1)     []
y(2)     [#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.network.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1}]
y(3)     #{listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.network.default.svc.cluster.local',parallelism=>1}
y(4)     {dict,7,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.network.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,7},port=>10200},undefined,<0.2870.0>}]],[],[],[],[[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,9},port=>10200},undefined,<0.2871.0>}]],[[#{listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,9},port=>10200},undefined,<0.2817.0>}]],[],[[#{listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,7},port=>10200},undefined,<0.2858.0>}]],[],[[#{listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@172-18-0-8.network-public.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,8},port=>10200},undefined,<0.2879.0>}]],[[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,6},port=>10200},undefined,<0.2874.0>}]],[[#{listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,6},port=>10200},undefined,<0.2867.0>}]],[],[],[]}}}
y(5)     {ok,[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.network.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1}],[{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},{membership_strategy,{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},{full_v1,<<3,9,124,143,228,132,107,38,117,180,132,96,78,58,196,73,251,74,235,183>>,{state_orset,[{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},[{<<36,183,99,119,228,252,140,50,251,114,173,171,46,82,11,195,104,167,24,239>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},[{<<64,255,155,107,248,239,243,209,167,181,155,250,251,204,223,174,239,38,178,214>>,false},{<<206,20,203,191,225,105,243,52,8,223,108,72,35,150,254,17,77,22,147,7>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},[{<<3,117,31,61,39,236,248,32,58,249,236,108,23,221,16,40,24,211,194,45>>,false},{<<8,107,22,133,203,80,255,134,118,5,120,40,208,34,113,69,87,16,55,73>>,true},{<<16,112,24,167,203,219,94,8,235,247,220,214,109,28,45,33,237,93,86,45>>,false},{<<36,76,0,101,147,122,94,43,98,114,238,174,171,89,196,143,66,247,31,102>>,false},{<<72,231,207,33,241,184,50,220,207,223,41,88,174,203,159,231,163,13,201,107>>,false},{<<98,50,57,29,96,143,9,22,173,51,33,3,81,13,247,115,165,85,236,35>>,false},{<<173,165,248,34,154,96,199,210,126,91,205,156,97,36,102,86,168,94,139,23>>,false},{<<231,30,161,230,99,90,37,133,149,137,192,94,16,194,231,222,204,25,193,113>>,false}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},[{<<7,72,158,14,65,227,17,97,247,71,210,186,106,72,36,177,159,116,115,208>>,true},{<<119,3,170,91,121,247,104,89,238,212,19,34,37,122,240,46,230,63,249,196>>,false}]}]}}}}},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},{membership_strategy,{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},{full_v1,<<3,9,124,143,228,132,107,38,117,180,132,96,78,58,196,73,251,74,235,183>>,{state_orset,[{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},[{<<36,183,99,119,228,252,140,50,251,114,173,171,46,82,11,195,104,167,24,239>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},[{<<64,255,155,107,248,239,243,209,167,181,155,250,251,204,223,174,239,38,178,214>>,false},{<<206,20,203,191,225,105,243,52,8,223,108,72,35,150,254,17,77,22,147,7>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},[{<<3,117,31,61,39,236,248,32,58,249,236,108,23,221,16,40,24,211,194,45>>,false},{<<8,107,22,133,203,80,255,134,118,5,120,40,208,34,113,69,87,16,55,73>>,true},{<<16,112,24,167,203,219,94,8,235,247,220,214,109,28,45,33,237,93,86,45>>,false},{<<36,76,0,101,147,122,94,43,98,114,238,174,171,89,196,143,66,247,31,102>>,false},{<<72,231,207,33,241,184,50,220,207,223,41,88,174,203,159,231,163,13,201,107>>,false},{<<98,50,57,29,96,143,9,22,173,51,33,3,81,13,247,115,165,85,236,35>>,false},{<<173,165,248,34,154,96,199,210,126,91,205,156,97,36,102,86,168,94,139,23>>,false},{<<231,30,161,230,99,90,37,133,149,137,192,94,16,194,231,222,204,25,193,113>>,false}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},[{<<7,72,158,14,65,227,17,97,247,71,210,186,106,72,36,177,159,116,115,208>>,true},{<<119,3,170,91,121,247,104,89,238,212,19,34,37,122,240,46,230,63,249,196>>,false}]}]}}}}},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},{membership_strategy,{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},{full_v1,<<3,9,124,143,228,132,107,38,117,180,132,96,78,58,196,73,251,74,235,183>>,{state_orset,[{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},[{<<36,183,99,119,228,252,140,50,251,114,173,171,46,82,11,195,104,167,24,239>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},[{<<64,255,155,107,248,239,243,209,167,181,155,250,251,204,223,174,239,38,178,214>>,false},{<<206,20,203,191,225,105,243,52,8,223,108,72,35,150,254,17,77,22,147,7>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},[{<<3,117,31,61,39,236,248,32,58,249,236,108,23,221,16,40,24,211,194,45>>,false},{<<8,107,22,133,203,80,255,134,118,5,120,40,208,34,113,69,87,16,55,73>>,true},{<<16,112,24,167,203,219,94,8,235,247,220,214,109,28,45,33,237,93,86,45>>,false},{<<36,76,0,101,147,122,94,43,98,114,238,174,171,89,196,143,66,247,31,102>>,false},{<<72,231,207,33,241,184,50,220,207,223,41,88,174,203,159,231,163,13,201,107>>,false},{<<98,50,57,29,96,143,9,22,173,51,33,3,81,13,247,115,165,85,236,35>>,false},{<<173,165,248,34,154,96,199,210,126,91,205,156,97,36,102,86,168,94,139,23>>,false},{<<231,30,161,230,99,90,37,133,149,137,192,94,16,194,231,222,204,25,193,113>>,false}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},[{<<7,72,158,14,65,227,17,97,247,71,210,186,106,72,36,177,159,116,115,208>>,true},{<<119,3,170,91,121,247,104,89,238,212,19,34,37,122,240,46,230,63,249,196>>,false}]}]}}}}},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},{membership_strategy,{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},{full_v1,<<3,9,124,143,228,132,107,38,117,180,132,96,78,58,196,73,251,74,235,183>>,{state_orset,[{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},[{<<36,183,99,119,228,252,140,50,251,114,173,171,46,82,11,195,104,167,24,239>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},[{<<64,255,155,107,248,239,243,209,167,181,155,250,251,204,223,174,239,38,178,214>>,false},{<<206,20,203,191,225,105,243,52,8,223,108,72,35,150,254,17,77,22,147,7>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},[{<<3,117,31,61,39,236,248,32,58,249,236,108,23,221,16,40,24,211,194,45>>,false},{<<8,107,22,133,203,80,255,134,118,5,120,40,208,34,113,69,87,16,55,73>>,true},{<<16,112,24,167,203,219,94,8,235,247,220,214,109,28,45,33,237,93,86,45>>,false},{<<36,76,0,101,147,122,94,43,98,114,238,174,171,89,196,143,66,247,31,102>>,false},{<<72,231,207,33,241,184,50,220,207,223,41,88,174,203,159,231,163,13,201,107>>,false},{<<98,50,57,29,96,143,9,22,173,51,33,3,81,13,247,115,165,85,236,35>>,false},{<<173,165,248,34,154,96,199,210,126,91,205,156,97,36,102,86,168,94,139,23>>,false},{<<231,30,161,230,99,90,37,133,149,137,192,94,16,194,231,222,204,25,193,113>>,false}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},[{<<7,72,158,14,65,227,17,97,247,71,210,186,106,72,36,177,159,116,115,208>>,true},{<<119,3,170,91,121,247,104,89,238,212,19,34,37,122,240,46,230,63,249,196>>,false}]}]}}}}}],{full_v1,<<3,9,124,143,228,132,107,38,117,180,132,96,78,58,196,73,251,74,235,183>>,{state_orset,[{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},[{<<36,183,99,119,228,252,140,50,251,114,173,171,46,82,11,195,104,167,24,239>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},[{<<64,255,155,107,248,239,243,209,167,181,155,250,251,204,223,174,239,38,178,214>>,false},{<<206,20,203,191,225,105,243,52,8,223,108,72,35,150,254,17,77,22,147,7>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},[{<<3,117,31,61,39,236,248,32,58,249,236,108,23,221,16,40,24,211,194,45>>,false},{<<8,107,22,133,203,80,255,134,118,5,120,40,208,34,113,69,87,16,55,73>>,true},{<<16,112,24,167,203,219,94,8,235,247,220,214,109,28,45,33,237,93,86,45>>,false},{<<36,76,0,101,147,122,94,43,98,114,238,174,171,89,196,143,66,247,31,102>>,false},{<<72,231,207,33,241,184,50,220,207,223,41,88,174,203,159,231,163,13,201,107>>,false},{<<98,50,57,29,96,143,9,22,173,51,33,3,81,13,247,115,165,85,236,35>>,false},{<<173,165,248,34,154,96,199,210,126,91,205,156,97,36,102,86,168,94,139,23>>,false},{<<231,30,161,230,99,90,37,133,149,137,192,94,16,194,231,222,204,25,193,113>>,false}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},[{<<7,72,158,14,65,227,17,97,247,71,210,186,106,72,36,177,159,116,115,208>>,true},{<<119,3,170,91,121,247,104,89,238,212,19,34,37,122,240,46,230,63,249,196>>,false}]}]}}}
y(6)     {state,<<3,9,124,143,228,132,107,38,117,180,132,96,78,58,196,73,251,74,235,183>>,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},[{'service_name@host2.network.default.svc.cluster.local',1}],[#{listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},#{listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},#{listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@172-18-0-8.network-public.default.svc.cluster.local',parallelism=>1}],[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1}],{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},[],{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},[],{dict,7,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,7},port=>10200},undefined,<0.2870.0>}]],[],[],[],[[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,9},port=>10200},undefined,<0.2871.0>}]],[[#{listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,9},port=>10200},undefined,<0.2817.0>}]],[],[[#{listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,7},port=>10200},undefined,<0.2858.0>}]],[],[[#{listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@172-18-0-8.network-public.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,8},port=>10200},undefined,<0.2879.0>}]],[[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,6},port=>10200},undefined,<0.2874.0>}]],[[#{listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,6},port=>10200},undefined,<0.2867.0>}]],[],[],[]}}},partisan_full_membership_strategy,{full_v1,<<3,9,124,143,228,132,107,38,117,180,132,96,78,58,196,73,251,74,235,183>>,{state_orset,[{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},[{<<36,183,99,119,228,252,140,50,251,114,173,171,46,82,11,195,104,167,24,239>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},[{<<64,255,155,107,248,239,243,209,167,181,155,250,251,204,223,174,239,38,178,214>>,false},{<<206,20,203,191,225,105,243,52,8,223,108,72,35,150,254,17,77,22,147,7>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},[{<<3,117,31,61,39,236,248,32,58,249,236,108,23,221,16,40,24,211,194,45>>,false},{<<8,107,22,133,203,80,255,134,118,5,120,40,208,34,113,69,87,16,55,73>>,true},{<<16,112,24,167,203,219,94,8,235,247,220,214,109,28,45,33,237,93,86,45>>,false},{<<36,76,0,101,147,122,94,43,98,114,238,174,171,89,196,143,66,247,31,102>>,false},{<<72,231,207,33,241,184,50,220,207,223,41,88,174,203,159,231,163,13,201,107>>,false},{<<98,50,57,29,96,143,9,22,173,51,33,3,81,13,247,115,165,85,236,35>>,false},{<<173,165,248,34,154,96,199,210,126,91,205,156,97,36,102,86,168,94,139,23>>,false},{<<231,30,161,230,99,90,37,133,149,137,192,94,16,194,231,222,204,25,193,113>>,false}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},[{<<7,72,158,14,65,227,17,97,247,71,210,186,106,72,36,177,159,116,115,208>>,true},{<<119,3,170,91,121,247,104,89,238,212,19,34,37,122,240,46,230,63,249,196>>,false}]}]}}}
y(7)     [#{listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.network.default.svc.cluster.local',parallelism=>1},#{listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@172-18-0-8.network-public.default.svc.cluster.local',parallelism=>1}]

0x00007f31523781a0 Return addr 0x00007f31536d0838 (gen_server:try_dispatch/4 + 152)
y(0)     []
y(1)     []
y(2)     []
y(3)     {state,<<3,9,124,143,228,132,107,38,117,180,132,96,78,58,196,73,251,74,235,183>>,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},[{'service_name@host2.network.default.svc.cluster.local',1}],[#{listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},#{listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},#{listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@172-18-0-8.network-public.default.svc.cluster.local',parallelism=>1}],[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1}],{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},[],{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},[],{dict,7,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,7},port=>10200},undefined,<0.2870.0>}]],[],[],[],[[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,9},port=>10200},undefined,<0.2871.0>}]],[[#{listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,9},port=>10200},undefined,<0.2817.0>}]],[],[[#{listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,7},port=>10200},undefined,<0.2858.0>}]],[],[[#{listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@172-18-0-8.network-public.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,8},port=>10200},undefined,<0.2879.0>}]],[[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,6},port=>10200},undefined,<0.2874.0>}]],[[#{listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,6},port=>10200},undefined,<0.2867.0>}]],[],[],[]}}},partisan_full_membership_strategy,{full_v1,<<3,9,124,143,228,132,107,38,117,180,132,96,78,58,196,73,251,74,235,183>>,{state_orset,[{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},[{<<36,183,99,119,228,252,140,50,251,114,173,171,46,82,11,195,104,167,24,239>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},[{<<64,255,155,107,248,239,243,209,167,181,155,250,251,204,223,174,239,38,178,214>>,false},{<<206,20,203,191,225,105,243,52,8,223,108,72,35,150,254,17,77,22,147,7>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},[{<<3,117,31,61,39,236,248,32,58,249,236,108,23,221,16,40,24,211,194,45>>,false},{<<8,107,22,133,203,80,255,134,118,5,120,40,208,34,113,69,87,16,55,73>>,true},{<<16,112,24,167,203,219,94,8,235,247,220,214,109,28,45,33,237,93,86,45>>,false},{<<36,76,0,101,147,122,94,43,98,114,238,174,171,89,196,143,66,247,31,102>>,false},{<<72,231,207,33,241,184,50,220,207,223,41,88,174,203,159,231,163,13,201,107>>,false},{<<98,50,57,29,96,143,9,22,173,51,33,3,81,13,247,115,165,85,236,35>>,false},{<<173,165,248,34,154,96,199,210,126,91,205,156,97,36,102,86,168,94,139,23>>,false},{<<231,30,161,230,99,90,37,133,149,137,192,94,16,194,231,222,204,25,193,113>>,false}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},[{<<7,72,158,14,65,227,17,97,247,71,210,186,106,72,36,177,159,116,115,208>>,true},{<<119,3,170,91,121,247,104,89,238,212,19,34,37,122,240,46,230,63,249,196>>,false}]}]}}}
y(4)     {connected,#{listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.network.default.svc.cluster.local',parallelism=>1},undefined,{full_v1,<<255,75,0,43,227,142,152,170,59,156,30,79,100,43,240,56,18,68,91,111>>,{state_orset,[{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},[{<<36,183,99,119,228,252,140,50,251,114,173,171,46,82,11,195,104,167,24,239>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},[{<<64,255,155,107,248,239,243,209,167,181,155,250,251,204,223,174,239,38,178,214>>,false},{<<206,20,203,191,225,105,243,52,8,223,108,72,35,150,254,17,77,22,147,7>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},[{<<3,117,31,61,39,236,248,32,58,249,236,108,23,221,16,40,24,211,194,45>>,false},{<<16,112,24,167,203,219,94,8,235,247,220,214,109,28,45,33,237,93,86,45>>,false},{<<36,76,0,101,147,122,94,43,98,114,238,174,171,89,196,143,66,247,31,102>>,false},{<<72,231,207,33,241,184,50,220,207,223,41,88,174,203,159,231,163,13,201,107>>,false},{<<98,50,57,29,96,143,9,22,173,51,33,3,81,13,247,115,165,85,236,35>>,false},{<<173,165,248,34,154,96,199,210,126,91,205,156,97,36,102,86,168,94,139,23>>,false},{<<231,30,161,230,99,90,37,133,149,137,192,94,16,194,231,222,204,25,193,113>>,false}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},[{<<7,72,158,14,65,227,17,97,247,71,210,186,106,72,36,177,159,116,115,208>>,true},{<<119,3,170,91,121,247,104,89,238,212,19,34,37,122,240,46,230,63,249,196>>,false}]}]}}}
y(5)     handle_info
y(6)     partisan_pluggable_peer_service_manager
y(7)     Catch 0x00007f31536d0888 (gen_server:try_dispatch/4 + 232)

0x00007f31523781e8 Return addr 0x00007f31536d16d8 (gen_server:handle_msg/6 + 1696)
y(0)     infinity
y(1)     partisan_pluggable_peer_service_manager
y(2)     {state,<<3,9,124,143,228,132,107,38,117,180,132,96,78,58,196,73,251,74,235,183>>,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},[{'service_name@host2.network.default.svc.cluster.local',1}],[#{listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},#{listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},#{listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@172-18-0-8.network-public.default.svc.cluster.local',parallelism=>1}],[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1}],{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},[],{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},[],{dict,7,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,7},port=>10200},undefined,<0.2870.0>}]],[],[],[],[[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,9},port=>10200},undefined,<0.2871.0>}]],[[#{listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,9},port=>10200},undefined,<0.2817.0>}]],[],[[#{listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,7},port=>10200},undefined,<0.2858.0>}]],[],[[#{listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@172-18-0-8.network-public.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,8},port=>10200},undefined,<0.2879.0>}]],[[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,6},port=>10200},undefined,<0.2874.0>}]],[[#{listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},{#{ip=>{172,18,0,6},port=>10200},undefined,<0.2867.0>}]],[],[],[]}}},partisan_full_membership_strategy,{full_v1,<<3,9,124,143,228,132,107,38,117,180,132,96,78,58,196,73,251,74,235,183>>,{state_orset,[{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},[{<<36,183,99,119,228,252,140,50,251,114,173,171,46,82,11,195,104,167,24,239>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},[{<<64,255,155,107,248,239,243,209,167,181,155,250,251,204,223,174,239,38,178,214>>,false},{<<206,20,203,191,225,105,243,52,8,223,108,72,35,150,254,17,77,22,147,7>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},[{<<3,117,31,61,39,236,248,32,58,249,236,108,23,221,16,40,24,211,194,45>>,false},{<<8,107,22,133,203,80,255,134,118,5,120,40,208,34,113,69,87,16,55,73>>,true},{<<16,112,24,167,203,219,94,8,235,247,220,214,109,28,45,33,237,93,86,45>>,false},{<<36,76,0,101,147,122,94,43,98,114,238,174,171,89,196,143,66,247,31,102>>,false},{<<72,231,207,33,241,184,50,220,207,223,41,88,174,203,159,231,163,13,201,107>>,false},{<<98,50,57,29,96,143,9,22,173,51,33,3,81,13,247,115,165,85,236,35>>,false},{<<173,165,248,34,154,96,199,210,126,91,205,156,97,36,102,86,168,94,139,23>>,false},{<<231,30,161,230,99,90,37,133,149,137,192,94,16,194,231,222,204,25,193,113>>,false}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},[{<<7,72,158,14,65,227,17,97,247,71,210,186,106,72,36,177,159,116,115,208>>,true},{<<119,3,170,91,121,247,104,89,238,212,19,34,37,122,240,46,230,63,249,196>>,false}]}]}}}
y(3)     partisan_pluggable_peer_service_manager
y(4)     <0.1357.0>
y(5)     {connected,#{listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.network.default.svc.cluster.local',parallelism=>1},undefined,{full_v1,<<255,75,0,43,227,142,152,170,59,156,30,79,100,43,240,56,18,68,91,111>>,{state_orset,[{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1},[{<<36,183,99,119,228,252,140,50,251,114,173,171,46,82,11,195,104,167,24,239>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},[{<<64,255,155,107,248,239,243,209,167,181,155,250,251,204,223,174,239,38,178,214>>,false},{<<206,20,203,191,225,105,243,52,8,223,108,72,35,150,254,17,77,22,147,7>>,true}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.default.svc.cluster.local',parallelism=>1},[{<<3,117,31,61,39,236,248,32,58,249,236,108,23,221,16,40,24,211,194,45>>,false},{<<16,112,24,167,203,219,94,8,235,247,220,214,109,28,45,33,237,93,86,45>>,false},{<<36,76,0,101,147,122,94,43,98,114,238,174,171,89,196,143,66,247,31,102>>,false},{<<72,231,207,33,241,184,50,220,207,223,41,88,174,203,159,231,163,13,201,107>>,false},{<<98,50,57,29,96,143,9,22,173,51,33,3,81,13,247,115,165,85,236,35>>,false},{<<173,165,248,34,154,96,199,210,126,91,205,156,97,36,102,86,168,94,139,23>>,false},{<<231,30,161,230,99,90,37,133,149,137,192,94,16,194,231,222,204,25,193,113>>,false}]},{#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},[{<<7,72,158,14,65,227,17,97,247,71,210,186,106,72,36,177,159,116,115,208>>,true},{<<119,3,170,91,121,247,104,89,238,212,19,34,37,122,240,46,230,63,249,196>>,false}]}]}}}

0x00007f3152378220 Return addr 0x00007f3158e653d0 (proc_lib:init_p_do_apply/3 + 72)
y(0)     []
y(1)     []
y(2)     Catch 0x00007f3158e653e8 (proc_lib:init_p_do_apply/3 + 96)

0x00007f3152378240 Return addr 0x000055e3ce117bb8 (<terminate process normally>)

Process 1410 is partisan_peer_service_events
backtrace:

Program counter: 0x00007f3153b3aa78 (gen:do_call/4 + 336)
0x00007f31516b06a0 []
0x00007f31516b06a8 infinity
0x00007f31516b06b0 []
0x00007f31516b06b8 #Ref<0.2459672142.441188356.233525>
0x00007f31516b06c0 <0.1482.0>

0x00007f31516b06c8 Return addr 0x00007f31536cc608 (gen_server:call/3 + 128)
y(0)     infinity
y(1)     {declare_dynamic,{<<"_membership">>,lwwregister},lwwregister}
y(2)     lasp_distribution_backend
y(3)     Catch 0x00007f31536cc608 (gen_server:call/3 + 128)

0x00007f31516b06f0 Return addr 0x00007f3151a56c20 (lasp_membership:update_membership/2 + 1032)
y(0)     []
y(1)     []
y(2)     []
y(3)     []
y(4)     []
y(5)     <<118,19,156,189,20,175,97,33,29,169,48,224,253,193,67,22,33,202,214,121>>
y(6)     [#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.network.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1}]

0x00007f31516b0730 Return addr 0x00007f315128cb28 (partisan_peer_service_events:handle_event/2 + 184)
y(0)     {state,#Fun<lasp_membership.0.12398551>}

0x00007f31516b0740 Return addr 0x00007f3153b42400 (gen_event:server_update/4 + 208)
y(0)     {state,#Fun<lasp_membership.0.12398551>}
y(1)     partisan_peer_service_events
y(2)     partisan_peer_service_events
y(3)     {update,[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.network.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1}]}
y(4)     {handler,partisan_peer_service_events,#Ref<0.2459672142.441188353.232392>,{state,#Fun<lasp_membership.0.12398551>},<0.1488.0>}
y(5)     Catch 0x00007f3153b42400 (gen_event:server_update/4 + 208)

0x00007f31516b0778 Return addr 0x00007f3153b42130 (gen_event:server_notify/4 + 128)
y(0)     []
y(1)     [{handler,partisan_peer_service_events,#Ref<0.2459672142.441188354.233273>,{state,fun plumtree_broadcast:update/1},<0.1430.0>},{handler,partisan_peer_service_events,#Ref<0.2459672142.441188354.233216>,{state,fun partisan_plumtree_broadcast:update/1},<0.1357.0>}]
y(2)     partisan_peer_service_events
y(3)     handle_event
y(4)     {update,[#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,8},port=>10200}],name=>'service_name@host2.network.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,7},port=>10200}],name=>'service_name@host1.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,9},port=>10200}],name=>'service_name@host3.default.svc.cluster.local',parallelism=>1},#{channels=>[undefined],listen_addrs=>[#{ip=>{172,18,0,6},port=>10200}],name=>'service_name@host0.default.svc.cluster.local',parallelism=>1}]}

0x00007f31516b07a8 Return addr 0x00007f3153b3fa80 (gen_event:handle_msg/6 + 240)
y(0)     {<0.2806.0>,#Ref<0.2459672142.441188356.233524>}
y(1)     []
y(2)     infinity
y(3)     partisan_peer_service_events
y(4)     <0.1357.0>

0x00007f31516b07d8 Return addr 0x00007f3158e653d0 (proc_lib:init_p_do_apply/3 + 72)
y(0)     []
y(1)     []
y(2)     Catch 0x00007f3158e653e8 (proc_lib:init_p_do_apply/3 + 96)

And this last process calls the first in the above list. I'm going to follow the code and try to figure out exactly how this happens, so I plan to update this issue when I have more information.

[NAR]

Looking at the traces now I see that the lasp:query call in step 1 blocks because the lasp_distribution_backend process is already handling a different request - it's probably coming from a lasp:declare_dynamiccall.

[NAR]

I looked more at the traces and found the following call flows leading to this deadlock:

1. A partisan_peer_service_client receives a TCP(?) message on a socket in a handle_info function. It decodes the message into {state, Tag, LocalState}, calls handle_message which will send a {connected, ...} message to the partisan_pluggable_peer_service_manager process.
2. The partisan_pluggable_peer_service_manager receives the {connected, ...} message in the handle_info function, calls partisan_peer_service_events:update which will call gen_event:sync_notify and blocks here until the partisan_peer_service_events process handles the notification.
3. The partisan_peer_service_events process receives the update event in the handle_event function and will call lasp_membership:update_membership (that's the callback in the state), which in turn eventually calls lasp_distribution_backend:declare_dynamic and blocks here until the lasp_distribution_backend process handles the declaration.
4. The lasp_distribution_backend process receives the {declare_dynamic, ...} call in the handle_call function, eventually calls a lasp_storage_backend:get and blocks here until the lasp_storage_backend process handles the {get, ...} call. The problem is that the lasp_storage_backend process is blocked (see step 2 in the call flow below).

The other call flow:

1. The lasp_state_based_synchronization_backend receives a {state_sync, ...} message (I think it is received from a timer), calls the init_state_sync function which calls lasp_storage_backend:fold and it blocks here until the call returns.
2. The lasp_storage_backend process receives a {fold, Function, ...} request where Function is the function created in lasp_state_based_synchronization_backend:init_state_sync. Then eventually calls lasp_ets_storage_backend:fold with the same function and blocks here until the call returns.
3. The lasp_ets_storage_backend process receives the {fold, Function, ...} request and calls ets:fold(Function, ...) which will call the inline function from lasp_state_based_synchronization_backend:init_state_sync. This anonymous function calls lasp_synchronization_backend:send which will call partisan_pluggable_peer_service_manager:cast_message. Despite it's name this function ends up doing a gen_server:call on the partisan_pluggable_peer_service_manager process, but that's blocked (see step 2 in the above call flow).

So it looks like the race is between two flows: an external TCP packet triggers a partisan_pluggable_peer_service_manager -> lasp_storage_backend chain (see the first flow above), while a scheduled {state_sync, ...} message triggers a lasp_storage_backend -> partisan_pluggable_peer_service_manager chain (see the second flow above). If these happen about the same time, we have the deadlock.

I don't know what these flows supposed to do, my guess based on variable and function names is that both want to update some kind of internal state, who are in the cluster. I looked into the 'service_name@host0.network.default.svc.cluster.local' ets table in a different run and saw this:

[{{<<"_membership">>,lwwregister},
  {dv,{state_lwwregister,{1611854259,
                          ['service_name@host1.network.default.svc.cluster.local',
                           'service_name@host3.network.default.svc.cluster.local',
                           'service_name@host0.network.default.svc.cluster.local']}},
      [],[],[],lwwregister,
      [{clock,[{<<112,81,31,35,249,226,203,99,138,234,42,146,124,20,143,158,12,
                  185,230,4>>,
                9}]},
       {dynamic,true}],
      0,[],[]}}]

so it looks line host2 is missing (maybe that's being added).