PubSub subscriber closes connection but remains running when messages are not acked/nacked #3646
Description
I have a suscriber which receives messages and uses the data to call an api. I had an issue with my api which left the connection hanging, and as such the subscription never acked/nacked the message. At this point, after a couple of minutes, the subscriber eventually closes its connection and the backlog begins to climb - when I patched the api to resolve the issue, the subscriber didn't process any old or new messages; however, when I restarted the application and create a new subscriber than the entire backlog is processed.
I can solve the issue in my api, and add some timeout logic to my receiver to stop this from occurring in the first place - but I am more interested in figuring out how to fail gracefully in an event like this where the subscriber is hanging. I have some logic in my application which periodically polls the subscriber to check that it is in a running state and restarts (stopAsync() + create a new subscription) if not. I can easily recreate this locally by adding a long delay to my API, and when I do so I notice again that the subscriber stops receiving any messages (I also checked the netstats and saw that there were no open connections to pubsub) - yet it still claims to be in a running state. Ideally my program would be able to identify that its not running and create a new subscriber (perhaps with some backoff), such that if I remove the delay in the api it would be automatically resolved without requiring me to manually restart the application.
Is there some way to detect that the subscriber is hanging? Is that the intended behavior for a subscriber when acks/nacks are delayed?