New deadlock in TransportSet and GrpcTimer

[RobinDevTW]

Hi,

I have encountered a new deadlock in TransportSet. I'm running under v1.0 with #2258 cherry-picked.

Found one Java-level deadlock:

"consumer-63":
waiting to lock monitor 0x00007f7ea408ca78 (object 0x0000000733ee5700, a java.lang.Object),
which is held by "consumer-35"
"consumer-35":
waiting to lock monitor 0x00007f7e9052a3c8 (object 0x000000070c6272f0, a java.lang.Object),
which is held by "grpc-timer-0"
"grpc-timer-0":
waiting to lock monitor 0x00007f7ea408ca78 (object 0x0000000733ee5700, a java.lang.Object),
which is held by "consumer-35"

Java stack information for the threads listed above:

"consumer-63":
at io.grpc.internal.DelayedClientTransport.newStream(DelayedClientTransport.java:118)

• waiting to lock <0x0000000733ee5700> (a java.lang.Object)
  at io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:214)
  at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
  at //.GrpcService$1$1.start(GrpcService.java:190)
  at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
  at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
  at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
  at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:135)

"consumer-35":
at io.grpc.internal.InUseStateAggregator.updateObjectInUse(InUseStateAggregator.java:50)

• waiting to lock <0x000000070c6272f0> (a java.lang.Object)
  at io.grpc.internal.TransportSet$BaseTransportListener.transportInUse(TransportSet.java:357)
  at io.grpc.internal.DelayedClientTransport.newStream(DelayedClientTransport.java:128)
• locked <0x0000000733ee5700> (a java.lang.Object)
  at io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:214)
  at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
  at //.GrpcService$1$1.start(GrpcService.java:190)
  at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
  at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
  at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
  at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:135)

"grpc-timer-0":
at io.grpc.internal.DelayedClientTransport.hasPendingStreams(DelayedClientTransport.java:283)

• waiting to lock <0x0000000733ee5700> (a java.lang.Object)
  at io.grpc.internal.TransportSet$1EndOfCurrentBackoff.run(TransportSet.java:246)
• locked <0x000000070c6272f0> (a java.lang.Object)
  at io.grpc.internal.LogExceptionRunnable.run(LogExceptionRunnable.java:56)

[RobinDevTW]

It looks like that our deployed code was not using the deadlock fix. We have now fully migrated to 1.0.1.
We 'll see if it happens again, but it is possible that this issue is actually a duplicate of issue 2246