How do I handle the exception - System.AggregateException A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread.

[marafiq]

Describe the bug

Getting the below exception when sending message to client using Groups.
How do I avoid this or gracefully handle this?
Like below

public static void Notify(long Id, params string[] keys)
  {
      GetHubContext<CountsHub>()
          .Clients
          .Group(GroupKey.FormatWith(Id))
          .notify(GetCounts(Id, default, keys));
  }

Exceptions (if any)

System.InvalidOperationException: No service connection found when sending message
  File "AzureTransport.cs", line 106, in Task AzureTransport.Send(object value)
    throw new InvalidOperationException("No service connection found when sending message");
  File "/_/src/Microsoft.AspNet.SignalR.Core/Hubs/HubDispatcher.cs", line 235, col 21, in Task HubDispatcher.InvokeHubPipeline(IHub hub, IJsonValue[] parameterValues, MethodDescriptor methodDescriptor, HubRequest hubRequest, StateChangeTracker tracker)+(Task<object> task) => { } [1]
  File "/_/src/Microsoft.AspNet.SignalR.Core/TaskAsyncHelper.cs", line 1006, col 17, in TResult TaskAsyncHelper.RunWithPreservedCulture<T1, T2, TResult>(CulturePair preservedCulture, Func<T1, T2, TResult> func, T1 arg1, T2 arg2)
  File "/_/src/Microsoft.AspNet.SignalR.Core/TaskAsyncHelper.cs", line 1017, col 13, in TResult TaskAsyncHelper.RunWithPreservedCulture<T, TResult>(CulturePair preservedCulture, Func<T, TResult> func, T arg)
  File "/_/src/Microsoft.AspNet.SignalR.Core/TaskAsyncHelper.cs", line 1071, col 43, in Task TaskAsyncHelper.ContinueWithPreservedCulture(Task task, Action<Task> continuationAction, TaskContinuationOptions continuationOptions)+(Task<T> t) => { }
  ?, in void ContinuationResultTaskFromResultTask<TAntecedentResult, TResult>.InnerInvoke()
  ?, in void Task.Execute()

Further technical details

• Your Azure SignalR SDK version - 1.21.4
• ASP.NET 4.8
• Your SignalR Client SDK version - 2.4.1

[vicancy]

Need to fix, currently the code throws at

azure-signalr/src/Microsoft.Azure.SignalR.AspNet/Transports/AzureTransport.cs

Line 115 in 89f3930

_ = received(value);

[marafiq]

@vicancy After updating to 2.4.3 the rate of exception has increased, getting 2x exceptions compared to previous version 2.4.1.
Any workaround?

On some days exception happens more frequently when there are fewer active users.

[vicancy]

Hi @marafiq could you share with me your resource name (send me through lianwei<at>microsoft.com), let me check if I can get any hints from the service logs.

[marafiq]

@vicancy thank you. Sent you the information via email.

[vicancy]

Such exception happens when server connections are dropped and messages failed to be sent out.

With the resource provided, the resource metrics shows that the exception increasing is probably caused by high server load and server connections closed by "PingTimeout", suggest increasing the resource Unit to see if the issue is mitigated.

[marafiq]

Thanks @vicancy for looking into it. After scaling the instance of SignalR Service so that usage stays under 70% these exception are still happening.

With the resource provided, the resource metrics shows that the exception increasing is probably caused by high server load and server connections closed by "PingTimeout"

I do not know if these are related or not, but I am looking at the process memory dump and noticing high fragmentation in Gen0 caused by byte[] rooted in SignalR websocket.

Memory Dump Pinned Byte Arrays coming from SignalR in Gen0

634 MB free space in GEN0

1. 

2. Group by Byte[] - 150 and 97 objects are pinned

3. When looking into 150 objects of byte[] - each array is of size 49316 and is pinned by SignalR

I am suspecting that ping timeouts are failing perhaps due to GC pressure at times, most importantly whey there are so many pinned arrays in Gen0, is it expected? If not what can I do differently.

[marafiq]

With respect to exceptions happening at similar rate - last 3 days. Singal is using 4 units in Premium plan

And after the scale - Blue line is server load, and red is connection quota

[marafiq]

Interestingly all those bytes array contains similar pay load of bytes, I convert those to string. Is it due continuously checking signal r service capacity?
???status?0?clientcount?43?servercount?2643?capacity?30000

Is it somehow related to EndpointMetrics

I would like to note that we are using SignalR with 2.4.3 in .NET 4.8. The only thing which might be little different than normal usage is that we use same SignalR instance by logically seperating connection by mapping app.MapAzureSignalR($"{appName}" with app name. Thus there will be many application instances with diff apps connected to it.

[vicancy]

Hi @marafiq, what is the memory/CPU usage of your app server instance? If the issue happens after upgrading to 2.4.3 SignalR, it might not be related to these G0 arrays, they are not related to 2.4.3 SignalR. Maybe SignalR 2.4.3 assumes more memory and adds memory pressure.

Under the hood, an {appName} introduce in (1+N)*[InitialHubServerConnectionCount,MaxHubServerConnectionCount]) server connections, N is the number of hubs, and introduce in (1+N) such ping messages. If you have M different app names, there would be M*(1+N) ping checks, so I think these 150 pinned array objects are the messages received from the service. (It is indeed something to improve in the SignalR service side, to reduce the frequency of sending such redundant info)

What is the current InitialHubServerConnectionCount and MaxHubServerConnectionCount set in your ServiceOptions(In old SDK the option is called ServerConnectionCount)? If your client connections and traffic is not large, maybe decreasing the count from default 5 to a smaller number 3 might help decrease the memory usage.

[marafiq]

Thank you @vicancy

Let me try to answer some of questions asked above. I think overall memory usage (generally memory pressure) and invalid opertion exceptions are interlinked. If you think these are two separate issues, should I open another issue to fix the memory pinning problems.

Is upgrade to 2.4.3 a problem?

Not entirely true. The number of errors has increased since upgrade, but these errors has been a problem from long time in diff versions. since I logged the error last year. Or its simply because our users have increase over time.

InitialHubServerConnectionCount and Max connections?

I am using default which is 5+1 initial connections per hub. We have 20 hubs. I have tried decreasing this number which introduce another problem of 503 errors. Since connections are scaled up or down by the SDK thus starting low would not really help much when the usage is high.

Resource Usage in App Service?

We are using Premium v3 P2V3.

Memory & CPU Usage are Healthy

The memory usage will certainly be better if there is less pinning. We are using high memory instance because sometime memory can jump around really bad, and fragmentation is big part of it.

150 pinned arrays - Loosing 600 MB due to Fragmentation in GEN0

Yes, I understand the number because we have many hubs. Reducing the number of hubs is one option. But pinning of these arrays does not help with overall app stability. Is it something which can be solved by using arrays from pool in the SDK itself.

But why these arrays are of size 49316 for end point metrics? Those arrays do not seem to use for messaging at all based on their content.
Another set of pinned arrays are created for each web socket, should these be using buffer from memory pool?

97 Pinned Arrays in GEN0

[vicancy]

Thanks for analyzing the issue @marafiq, yes I agree with you that the memory issue is the root issue. It might be related to this different logic in WebSocketsTransport. Please allow me to look into the issue to see how to improve that.

azure-signalr/src/Microsoft.Azure.SignalR.Common/ServiceConnections/Internal/WebSocketsTransport.cs

Line 293 in ec579ce

var isArray = MemoryMarshal.TryGetArray<byte>(memory, out var arraySegment);

I am using default which is 5+1 initial connections per hub. We have 20 hubs. I have tried decreasing this number which introduce another problem of 503 errors. Since connections are scaled up or down by the SDK thus starting low would not really help much when the usage is high.

For this, setting MaxHubServerConnectionCount controls the max count the connections could scale up to.