Deadlock when cancelling active call while disposing channel #2206
Description
What version of gRPC and what language are you using?
C#, using Grpc.Net.Client version 2.55.0
What operating system (Linux, Windows,...) and version?
Windows 10.0.19042
What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)
.NET 6.0.400
What did you do?
We have a streaming gRPC call between two windows services which runs for the duration of the client service. When the client service is stopped, the call is canceled and the grpc client is disposed.
We have recently started seeing occasional hangs when the client is shutting down. The symptoms appear very similar to #2119, however I've confirmed that we are using version 2.55.0 of the client which should be fixed.
Debugging into the hung service, the cancellation callback for the service stop cancellation token seems to be stuck waiting for a lock in Grpc.Net.Client. I think the line numbers are referring to decompiled code; this line is what visual studio points at for me.
Stack trace for thread 219848
[Waiting on lock owned by Thread 213652, double-click or press enter to switch to thread]
ntdll.dll!NtWaitForMultipleObjects�() Unknown
KERNELBASE.dll!00007ffb63f41b40() Unknown
[Managed to Native Transition]
Grpc.Net.Client.dll!Grpc.Net.Client.Internal.Retry.RetryCallBase<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.ClearRetryBuffer() Line 422 C#
Grpc.Net.Client.dll!Grpc.Net.Client.Internal.Retry.RetryCallBase<System.__Canon, System.__Canon>.Cleanup() Line 397 C#
Grpc.Net.Client.dll!Grpc.Net.Client.Internal.Retry.RetryCall<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.StartRetry(System.Action<Grpc.Net.Client.Internal.GrpcCall<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>> startCallFunc) Line 185 C#
[Resuming Async Method]
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Grpc.Net.Client.Internal.Retry.RetryCall<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.<StartRetry>d__7>.ExecutionContextCallback(object s) Line 108 C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 137 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Grpc.Net.Client.Internal.Retry.RetryCall<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.<StartRetry>d__7>.MoveNext(System.Threading.Thread threadPoolThread) Line 134 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Grpc.Net.Client.Internal.Retry.RetryCall<System.__Canon, System.__Canon>.<StartRetry>d__7>.MoveNext() Line 118 C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox box, bool allowInlining) Line 198 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2693 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task<Grpc.Core.Status>.TrySetResult(Grpc.Core.Status result) Line 176 C#
System.Private.CoreLib.dll!System.Threading.Tasks.TaskCompletionSource<Grpc.Core.Status>.TrySetResult(Grpc.Core.Status result) Line 227 C#
Grpc.Net.Client.dll!Grpc.Net.Client.Internal.GrpcCall<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.CancelCall(Grpc.Core.Status status) Line 330 C#Grpc.Net.Client.dll!Grpc.Net.Client.Internal.GrpcCall<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.CancelCallFromCancellationToken(System.Threading.CancellationToken cancellationToken) Line 323 C#
Grpc.Net.Client.dll!Grpc.Net.Client.Internal.GrpcCall<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.RegisterCancellation.AnonymousMethod__69_0(object state, System.Threading.CancellationToken ct) Line 316 C#
System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.Invoke(System.Delegate d, object state, System.Threading.CancellationTokenSource source) Line 688 C#
System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.CallbackNode.ExecuteCallback.AnonymousMethod__9_0(object s) Line 264 C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 137 C#
System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.CallbackNode.ExecuteCallback() Line 260 C#
System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(bool throwOnFirstException) Line 621 C#
System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.NotifyCancellation(bool throwOnFirstException) Line 568 C#
System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.Cancel() Line 365 C#
Microsoft.Extensions.Hosting.Abstractions.dll!Microsoft.Extensions.Hosting.BackgroundService.StopAsync(System.Threading.CancellationToken cancellationToken) Line 34 C#
Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.Internal.Host.StopAsync(System.Threading.CancellationToken cancellationToken) Line 114 C#
Microsoft.Extensions.Hosting.Abstractions.dll!Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.StopAsync(Microsoft.Extensions.Hosting.IHost host, System.TimeSpan timeout) Line 18 C#
Spectrum.Site.Common.dll!Spectrum.Site.Common.WindowsServiceHost.WaitForStop() Line 97 C#
Spectrum.Site.Common.dll!Spectrum.Site.Common.WindowsServiceHost.OnStop() Line 83 C#
System.ServiceProcess.ServiceController.dll!System.ServiceProcess.ServiceBase.DeferredStop() Line 410 C#
System.ServiceProcess.ServiceController.dll!System.ServiceProcess.ServiceBase.ServiceCommandCallback.AnonymousMethod__2(object _) Line 638 C#
System.Private.CoreLib.dll!System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute() Line 21 C#
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 541 C#
System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 1022 C#
System.Private.CoreLib.dll!System.Threading.Thread.StartCallback() Line 392 C#
[Native to Managed Transition]
kernel32.dll!00007ffb65067614() Unknown
ntdll.dll!RtlUserThreadStart�() Unknown
The lock is being held by another thread, 213652. The lock was acquired here. Further down the callstack, it is stuck waiting on a call to CancellationTokenRegistration.Dispose here which apparently has to wait for all active callbacks to finish. The callback it is waiting on is Callback = {Method = {Void <RegisterCancellation>b__69_0(System.Object, System.Threading.CancellationToken)}}, CallbackState = {Grpc.Net.Client.Internal.GrpcCall<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>}, which appears in the first thread, completing the deadlock.
Stack trace for thread 213652
ntdll.dll!NtDelayExecution�() Unknown
KERNELBASE.dll!00007ffb63f3b44e() Unknown
[Managed to Native Transition]
System.Private.CoreLib.dll!System.Threading.Thread.Sleep(int millisecondsTimeout) Line 667 C#
System.Private.CoreLib.dll!System.Threading.SpinWait.SpinOnceCore(int sleep1Threshold) Line 57 C#
System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.Registrations.WaitForCallbackToComplete(long id) Line 182 C#
System.Private.CoreLib.dll!System.Threading.CancellationTokenRegistration.Dispose() Line 36 C#
Grpc.Net.Client.dll!Grpc.Net.Client.Internal.GrpcCall<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.Cleanup(Grpc.Core.Status status) Line 191 C#
Grpc.Net.Client.dll!Grpc.Net.Client.Internal.GrpcCall<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.Dispose() Line 172 C#
Grpc.Net.Client.dll!Grpc.Net.Client.Internal.Retry.RetryCallBase<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.Dispose(bool disposing) Line 386 C#
Grpc.Net.Client.dll!Grpc.Net.Client.Internal.Retry.RetryCall<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.Dispose(bool disposing) Line 215 C#
Grpc.Net.Client.dll!Grpc.Net.Client.Internal.Retry.RetryCallBase<System.__Canon, System.__Canon>.Dispose() Line 131 C#
Grpc.Net.Client.dll!Grpc.Net.Client.Internal.HttpClientCallInvoker.Callbacks<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>..cctor.AnonymousMethod__4_3(object state) Line 21 C#
Spectrum.Site.Common.dll!Spectrum.Site.Common.SyncService.SyncServiceClient.GetSubscriptions(bool waitForProcessing, System.Threading.CancellationToken cancellationToken) Line 51 C#
[Resuming Async Method]
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<System.__Canon>.ExecutionContextCallback(object s) Line 108 C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 137 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Spectrum.Site.Common.SyncService.SyncServiceClient.<GetSubscriptions>d__4>.MoveNext(System.Threading.Thread threadPoolThread) Line 134 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<System.__Canon>.MoveNext() Line 118 C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox box, bool allowInlining) Line 198 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2748 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishStageThree() Line 1966 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishStageTwo() Line 1954 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishSlow(bool userDelegateExecute) Line 1899 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.TrySetException(object exceptionObject) Line 2574 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<bool>.SetException(System.Exception exception, ref System.Threading.Tasks.Task<bool> taskField) Line 363 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<bool>.SetException(System.Exception exception) Line 353 C#
[Completed] Grpc.Net.Client.dll!Grpc.Net.Client.Internal.Retry.RetryCallBaseClientStreamReader<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.MoveNext(System.Threading.CancellationToken cancellationToken) Line 31 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<bool>.AsyncStateMachineBox<Grpc.Net.Client.Internal.Retry.RetryCallBaseClientStreamReader<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.<MoveNext>d__4>.ExecutionContextCallback(object s) Line 108 C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 137 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<bool>.AsyncStateMachineBox<Grpc.Net.Client.Internal.Retry.RetryCallBaseClientStreamReader<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.<MoveNext>d__4>.MoveNext(System.Threading.Thread threadPoolThread) Line 134 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<bool>.AsyncStateMachineBox<Grpc.Net.Client.Internal.Retry.RetryCallBaseClientStreamReader<System.__Canon, System.__Canon>.<MoveNext>d__4>.MoveNext() Line 118 C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox box, bool allowInlining) Line 198 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2748 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishStageThree() Line 1966 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishStageTwo() Line 1954 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishSlow(bool userDelegateExecute) Line 1899 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.TrySetException(object exceptionObject) Line 2574 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<bool>.SetException(System.Exception exception, ref System.Threading.Tasks.Task<bool> taskField) Line 363 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<bool>.SetException(System.Exception exception) Line 353 C#
[Completed] Grpc.Net.Client.dll!Grpc.Net.Client.Internal.HttpContentClientStreamReader<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.MoveNextCore(System.Threading.CancellationToken cancellationToken) Line 190 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<bool>.AsyncStateMachineBox<Grpc.Net.Client.Internal.HttpContentClientStreamReader<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.<MoveNextCore>d__18>.ExecutionContextCallback(object s) Line 108 C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 137 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<bool>.AsyncStateMachineBox<Grpc.Net.Client.Internal.HttpContentClientStreamReader<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.<MoveNextCore>d__18>.MoveNext(System.Threading.Thread threadPoolThread) Line 134 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<bool>.AsyncStateMachineBox<Grpc.Net.Client.Internal.HttpContentClientStreamReader<System.__Canon, System.__Canon>.<MoveNextCore>d__18>.MoveNext() Line 118 C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox box, bool allowInlining) Line 198 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2748 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Line 176 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.SetExistingTaskResult(System.Threading.Tasks.Task<System.__Canon> task, System.__Canon result) Line 340 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder<Spectrum.Site.SyncService.Grpc.SubscriptionList>.SetResult(Spectrum.Site.SyncService.Grpc.SubscriptionList result) Line 121 C#
[Completed] Grpc.Net.Client.dll!Grpc.Net.Client.Internal.StreamExtensions.ReadMessageAsync<Spectrum.Site.SyncService.Grpc.SubscriptionList>(System.IO.Stream responseStream, Grpc.Net.Client.Internal.GrpcCall call, System.Func<Grpc.Core.DeserializationContext, Spectrum.Site.SyncService.Grpc.SubscriptionList> deserializer, string grpcEncoding, bool singleMessage, System.Threading.CancellationToken cancellationToken) Line 128 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<Spectrum.Site.SyncService.Grpc.SubscriptionList>.AsyncStateMachineBox<Grpc.Net.Client.Internal.StreamExtensions.<ReadMessageAsync>d__4<Spectrum.Site.SyncService.Grpc.SubscriptionList>>.ExecutionContextCallback(object s) Line 108 C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 137 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<Spectrum.Site.SyncService.Grpc.SubscriptionList>.AsyncStateMachineBox<Grpc.Net.Client.Internal.StreamExtensions.<ReadMessageAsync>d__4<Spectrum.Site.SyncService.Grpc.SubscriptionList>>.MoveNext(System.Threading.Thread threadPoolThread) Line 134 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Grpc.Net.Client.Internal.StreamExtensions.<ReadMessageAsync>d__4<System.__Canon>>.MoveNext() Line 118 C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox box, bool allowInlining) Line 198 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2748 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task<int>.TrySetResult(int result) Line 176 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<int>.SetExistingTaskResult(System.Threading.Tasks.Task<int> task, int result) Line 340 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder<int>.SetResult(int result) Line 121 C#
[Completed] System.Net.Http.dll!System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(System.Memory<byte> buffer, System.Net.Http.HttpResponseMessage responseMessage, System.Threading.CancellationToken cancellationToken) Unknown
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<int>.AsyncStateMachineBox<System.Net.Http.Http2Connection.Http2Stream.<ReadDataAsync>d__69>.ExecutionContextCallback(object s) Line 108 C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread threadPoolThread, System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 191 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<int>.AsyncStateMachineBox<System.Net.Http.Http2Connection.Http2Stream.<ReadDataAsync>d__69>.MoveNext(System.Threading.Thread threadPoolThread) Line 140 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<int>.AsyncStateMachineBox<System.Net.Http.Http2Connection.Http2Stream.<ReadDataAsync>d__69>.ExecuteFromThreadPool(System.Threading.Thread threadPoolThread) Line 113 C#
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 569 C#
System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 1022 C#
System.Private.CoreLib.dll!System.Threading.Thread.StartCallback() Line 392 C#
[Native to Managed Transition]
kernel32.dll!00007ffb65067614() Unknown
ntdll.dll!RtlUserThreadStart�() Unknown
I don't know if its significant, but a third thread is also trying to call CancellationTokenRegistration.Dispose and seems to be waiting on the same callback. As far as I can tell its not holding any locks or blocking the other two threads though.
Stack trace for thread 242260
ntdll.dll!NtDelayExecution�() Unknown
KERNELBASE.dll!00007ffb63f3b44e() Unknown
[Managed to Native Transition]
System.Private.CoreLib.dll!System.Threading.Thread.Sleep(int millisecondsTimeout) Line 667 C#
System.Private.CoreLib.dll!System.Threading.SpinWait.SpinOnceCore(int sleep1Threshold) Line 57 C#
System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.Registrations.WaitForCallbackToComplete(long id) Line 182 C#
System.Private.CoreLib.dll!System.Threading.CancellationTokenRegistration.Dispose() Line 36 C#
Grpc.Net.Client.dll!Grpc.Net.Client.Internal.GrpcCall<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.Cleanup(Grpc.Core.Status status) Line 191 C#
Grpc.Net.Client.dll!Grpc.Net.Client.Internal.GrpcCall<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.RunCall(System.Net.Http.HttpRequestMessage request, System.TimeSpan? timeout) Line 453 C#
[Resuming Async Method]
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Grpc.Net.Client.Internal.GrpcCall<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.<RunCall>d__73>.ExecutionContextCallback(object s) Line 108 C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread threadPoolThread, System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 191 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Grpc.Net.Client.Internal.GrpcCall<Google.Protobuf.WellKnownTypes.Empty, Spectrum.Site.SyncService.Grpc.SubscriptionList>.<RunCall>d__73>.MoveNext(System.Threading.Thread threadPoolThread) Line 140 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Grpc.Net.Client.Internal.GrpcCall<System.__Canon, System.__Canon>.<RunCall>d__73>.ExecuteFromThreadPool(System.Threading.Thread threadPoolThread) Line 113 C#
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 569 C#
System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 1022 C#
System.Private.CoreLib.dll!System.Threading.Thread.StartCallback() Line 392 C#
[Native to Managed Transition]
kernel32.dll!00007ffb65067614() Unknown
ntdll.dll!RtlUserThreadStart�() Unknown