Netty improperly delays sending SETTINGS ACK

[ejona86]

As seen in grpc/grpc#9956:

... it appears Java/Netty is doing the wrong thing. encoder.remoteSettings() is called before writeSettingsAck(). encoder.remoteSettings() calls flowController().initialWindowSize() which results in the flow controller writing pending bytes.

Two immediate changes should probably be made: 1) Send the settings ack immediately before processing the settings frame in the decoder. Any failure in processing would probably result in the connection being closed, so this doesn't seem dangerous. 2) Stop triggering writePendingBytes() in the flow controller and instead wait for the flush() like normal.

[ejona86]

Based on testing, it appears this was introduced during the upgrade from Netty 4.1.6 to Netty 4.1.8.

[ejona86]

Further testing showed that Netty 4.1.6 was also impacted (as the code would imply). Although it is hard to reproduce, we have seen failures from grpc-java v1.0.2 in addition to those already seen in v1.1.x. We think v1.0.2 is harder to trigger because it has less efficient flush behavior, as 9606995 is the first commit that readily causes the failure in our test environment. Simply adding a writeQueue.scheduleFlush() to NettyClientStream.request() greatly reduces the failure rate in our test environment.