[PERF] .NET6 Performance Regression

[Aaronontheweb]

Version Information
Version of Akka.NET? v1.4.28
Which Akka.NET Modules? Akka, Akka.Remote

Describe the performance issue
Our RemotePingPong benchmark has been the standard used for 7 years or so to measure throughput passing over a single Akka.Remote connection between two ActorSystem instances. It's a crucial benchmark because it measures the biggest bottleneck in Akka.Remote networks: the end to end response time over a single connection.

Over the lifespan of .NET Core since 2017 we've seen steady improvements in the benchmark numbers each time a new version of the .NET runtime is released usually as an improvement of underlying threading / concurrency / IO primitives introduced into the runtime itself.

With the release of .NET 6, however, we've noticed that while the overall throughput in some measures remains higher than on .NET 5 for these same reasons - there are steady, reproducible, long-lasting drops in total throughput that occur only on .NET 6.

Data and Specs
Here are the RemotePingPong numbers from my local development machine, a Gen 1 8-core Ryzen, on .NET Core 3.1:

Edit: update the .NET Core 3.1 benchmark numbers to include the settings from #5386

And here are the equivalent numbers for this same benchmark on .NET 6:

I've been able to reproduce this consistently - a sustained drop in throughput that lasts for roughly 30s. We've also noticed this in the Akka.NET test suite since merging in #5373 - the number of failures in the test suite has grown and has started to include tests that historically have not been racy. We've also observed this separately in the Phobos repository which we also upgraded to use the .NET 6 SDK.

There is definitely something amiss here with how Akka.NET runs on top of .NET 6.

Expected behavior
A consistent level of performance across all benchmarks.

Actual behavior
Intermittent lag, declines in throughput, and unexplained novel race conditions.

Environment
.NET 6, Windows

Additional context
There is some speculation from other members of the Akka.NET team that the issue could be related to some of the .NET ThreadPool and thread injection changes made in .NET 6:

• Improve the rate of thread injection for blocking due to sync-over-async dotnet/runtime#53471
• Enable the portable thread pool by default in coreclr dotnet/runtime#43841
• And a performance that was reported and later fixed pertaining to the new managed threadpool implementation: [Perf] Widespread regressions after threadpool change dotnet/runtime#44211

[to11mtm]

@Aaronontheweb I would suggest getting some more heruistics on the threads, i.e. set up multisampling (I mean, reading number of TP threads should be cheap overall) and figure out averageThreads, MaxThreads and threads over 95% of time. I can hack up a snippet to help if you'd like.

If I had to guess overall, I'd wager it's an issue with the changes in Threadpool hill-climbing algorithm in core, MAYBE some interplay with ThreadLocal if we are creating and destroying threads more frequently (not sure TBH, would need to dig.)

It may or may not be worth it to try running this with the actors running under DTP wherever possible under both environments; perf may be lower in both cases but that would at least help to isolate whether it -is- the .NET threadpool itself or something else creeping in.

[Aaronontheweb]

I can hack up a snippet to help if you'd like.

That would be very helpful!

It may or may not be worth it to try running this with the actors running under DTP wherever possible under both environments; perf may be lower in both cases but that would at least help to isolate whether it -is- the .NET threadpool itself or something else creeping in.

That's a good idea. This should be easy to configure.

@Zetanova noticed anything like this on .NET 6 with the ChannelDispatcher?

[to11mtm]

I can hack up a snippet to help if you'd like.

That would be very helpful!

Here you go: https://gist.github.com/to11mtm/1c3f5137a207d59d5f3e61bb198aeeae . Note I haven't exactly -tested- this with the way I'm abusing CTS, and it's not -quite- thread safe (i.e. you'd need to make changes to be able to observe values while it is monitoring.) The only thing you might want to tweak is adding a method to pre-allocate the array and minimize the chance of a GC Happening:

public void PresetContainer(TimeSpan interval, TimeSpan worstCase)
{
   ThreadStats = new List<int>( (int)(worstCase/interval)+1);
}

There's plenty of other opportunities for cleanup/niceness there, but it is a good start if you want to get fancier (i.e. track when timer fires noticeably off interval to maintain rate, which is an indication of complete system overpressure since we are on a dedicated thread in this loop.)

[Aaronontheweb]

Ah thanks, so do I just launch that in the background along with the benchmark to have it gather samples concurrently in its own thread?

[Zetanova]

Sorry, dont have exp. with .net6 yet,
I am always waiting few month after release.

For sequential workload the thread number are to high.
Optimal count would be:

Akka.Remote:
2x IO thread for network
1-3x Threads for processing and serialization

Akka.Actor:
8x (LP-Count) for ActorDispacher instances
0x No AsyncTasks inside the Actors

This makes a theoretical max total for your system of 11-13 threads for each node.

If more threads/tasks then the theoretical max total are scheduled
then the execution sequence will relay only on the external OS thread scheduler and .net TaskScheduler algo.
There algo's are "superior" to what we can make, but do work on a subset of information.
One critical property for them is missing.

That's way I made the https://github.com/Zetanova/Akka.Experimental.ChannelTaskScheduler
to get control of the sequence of execution and not to overload the TaskScheduler with mixed priority tasks
and task that could be delayed for "minutes".

The actor system workload has an unique task scheduling property to be able to indefinitely delay the execution of ActorCells.
If the whole workload and more then the physically possible concurrent count is queued inside a normal TaskScheduler at once
this property will be simply lost. The TaskScheduler algo will do its best to execute all tasks as fast as possible, but as mentioned above it is working with a subset of information.

To use System.Threading.Channels is not a requirement but it solves major problems:

1. thread-safe dequeue and enqueue operation in a performant way (and its managed)
2. it can use async task operations and utilize the regular dotnet TaskScheduler (no dedicated thread management required)
3. prioritizing of work items (system and cluster work items)
4. simple implementation

@Aaronontheweb
Long story short:

1. You can try to run the RemotePingPong benchmark with the ChannelDispatcher
   I am confidant that this issue will simply disappear.

2. You can make a simple test benchmark with a 2 node cluster where
   Node A sends a very high amount of workload to Node B
   Node B executes a fake Task.Deplay(msg.WorkSize).PipeTo(Sender, new WorkCompleted(msg.Id))
   The Test should run over a fixed period (3-5 minutes), the completed work items over this period should be counted
   and used as score/result
   What I am expecting is that the current/old scheduling system will produce a lot of cluster delay workings
   or even a node dissociation.

[Aaronontheweb]

You can try to run the RemotePingPong benchmark with the ChannelDispatcher
I am confidant that this issue will simply disappear.

I'll give that a shot @Zetanova and report back on here.

[Aaronontheweb]

Filed a CoreCLR issue here: dotnet/runtime#62967