Fully Adaptive Stateless Worker

[ledjon-behluli]

This PR introduced adaptive scale-down for stateless worker grains. As the "workers" of a StatelessWorker are regular grain, their deactivation currently is left to the activation collector. For on-going load this is not a problem, but for burst-like loads the workers sit idle in memory when they can be collected.

Adaptive scale-down is optional and does not incur overhead when not used.

Users can make use of it in 2 ways:

• Directly on the grain implementation:

[StatelessWorker(proactiveWorkerCollection: true)]
public class MyGrain : ITestGrain;

• Via global configuration:

siloBuilder.Configure<StatelessWorkerOptions>(o =>
{
    o.UseProactiveWorkerCollection = true; // Applies to all stateless worker grain
}

---

EDIT: 3/21/25

After some iterations the above statements are not valid anymore.

• The feature is now opt-out

• Directly on the grain implementation:

[StatelessWorker(removeIdleWorkers: false)] // (true - by default)
public class MyGrain : ITestGrain;

• Via global configuration:

siloBuilder.Configure<StatelessWorkerOptions>(o =>
{
    o.RemoveIdleWorkers = false; // Applies to all stateless worker grain (true - by default)
    o.IdleWorkersInspectionPeriod = TimeSpan.From(...);
    o.MinIdleCyclesBeforeRemoval = 3;
}

An in-depth explanation can be found here: https://www.ledjonbehluli.com/posts/orleans_adaptive_stateless_worker/

Microsoft Reviewers: Open in CodeFlow

[ledjon-behluli]

@ReubenBond I've address your comments, and regarding the unsynchronized access: I've handled it (please have a look), otherwise I've ideally would have chosen a ConcurrentDictionary<ActivationId, ActivationData> but we loose ordering with that when it comes to prioritizing early created workers. I've resorted to locks but only used where its absolutely necessary.

[ReubenBond]

@ledjon-behluli does this branch need a push? The build is failing due to:

Error: /home/runner/work/orleans/orleans/test/TesterInternal/General/GrainPlacementTests.cs(141,33): error CS7036: There is no argument given that corresponds to the required parameter 'removeIdleWorkers' of 'StatelessWorkerPlacement.StatelessWorkerPlacement(int, bool)' [/home/runner/work/orleans/orleans/test/TesterInternal/TesterInternal.csproj::TargetFramework=net8.0]

[ledjon-behluli]

@ReubenBond yes, please check discord on the thread regarding the opt-out and property-based approach. Let me know if you're ok with that

[ReubenBond]

LGTM - I added a question and pushed some changes. My advice about using a property for StatelessWorkerAttribute was bad - I've changed that in my commit.
Please take a look and let me know if you'd change anything.