Akka.Cluster: quarantining / reachability changes appear to be extremely sensitive

[Aaronontheweb]

Akka.NET version: 1.4.17

As the description states, in a cluster with the following composition:

~20 unreachable nodes
~40 weakly up nodes
~30 up nodes

Any attempts to Down the unreachable nodes successfully change their status from Up / WeaklyUp -> Down, but they are never correctly exited from the cluster nor are any of the WeaklyUp nodes promoted to Up. Using a split brain resolver and a pbm cluster down-unreachable command have both demonstrated this.

This looks like a gossip convergence issue to me - going to investigate.

[ismaelhamed]

Is this related to #4757 ? Every single time a DOWN does not seem to propagate is because there's at least one node in TERMINATED state.

[Aaronontheweb]

Changed the title to: "Akka.Cluster: quarantining / reachability changes appear to be extremely sensitive"

Looks to me like there's some issues, which can be easily reproduced during a large Akka.Cluster deployment (i.e. scaling from 3 to 25 nodes via kubectl), that really shouldn't be occurring. Message traffic isn't all that high when this occurs - I think it has to do with the failure-detector being extremely overtuned in Akka.Cluster.

[Aaronontheweb]

https://gist.github.com/Aaronontheweb/66095c9340437c0576cf55876d65c1f7 - some related theories I'm probing related to this

[Aaronontheweb]

Worked on this as part of a "heartbeats not setting sent fast enough" hypothesis #4882

[Aaronontheweb]

Working theories as to what is causing this:

• Addition of AppVersion data skews reachability detection when a "rolling upgrade" is detected
• Port scala akka PR #26816 to Akka.NET #4511 - introduction of the InternalDispatcher actually made heartbeat scheduling less reliable, especially on larger node rings
• Increase default number of nodes for cluster failure detection #4097 - that this change, which should have made it more difficult for the cluster to arrive at REACHABLE vs. UNREACHABLE decisions, doesn't really have any effect and any single node in the observability ring can effectively mark any of the others as unreachable.
• Bug in the cluster heartbeat system that causes it to be overly sensitive to early heartbeat data at startup. This would be an issue with the Phi Accrual system if that were the case.
• Pre-programmed latency between the arrival of Gossip notifying a node about multiple joining nodes vs. the delay in actually receiving incoming connections from those nodes. i.e. 3 up members in cluster, 12 new nodes join, all nodes have to monitor up to 9 other nodes, does the "first heartbeat" countdown start only after the Akka.Remote association goes through or before?

Some of these should be pretty easy to rule out by reading through the code - others are going to require a stress test that expands and contracts a cluster running inside Kubernetes, most likely.

[Aaronontheweb]

Ruled out the AppVersion changes last night - but wanted to include them for correctness. Some of the more recent JVM clustering code factors in AppVersion as part of reachability but none of that has been ported yet (I also think the way they're tracking versions throughout the cluster is probably wrong - seems like it should be done on a per-role basis, not a cluster-wide basis)

[to11mtm]

@Aaronontheweb Just for clarity, do we know if this is (1) a regression, or (2) an existing issue?

Reason I ask is if it's an existing issue and not a regression, one more theory:

• Schedulers operating at fixed rates rather than fixed delays; under fixed-rate (which is what we do currently), if there are GC pauses or heavy load (and frankly, on our cluster, starting up is the most CPU load we see on the app) this could lead to multiple heartbeats being sent out quickly, which ends up confusing the phi accrual.

Message traffic isn't all that high when this occurs - I think it has to do with the failure-detector being extremely overtuned in Akka.Cluster.

I will add, FWIW, that we had to de-tune a -lot- of settings for our application;

• Remoting Failure detector (99% sure, will check)
• Cluster Failure detector
• SBR Failure Detector (we're still on 1.4.14, fwiw)
• Coordinated-shutdown settings (this was a biiiig one for sharding stability, needed to give everything enough time to properly shut down)

I can try to find more specifics if it will help.