Monitoring system tests backoff is insufficient sometimes

[dhermes]

• https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/2744
• https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/2746

/cc @liyanhui1228 @duggelz

[liyanhui1228]

The max retries for the failed test is 10, but the tests failed after retry for 5 times, what stops it from retrying?

[dhermes]

1. 10 retries is probably too many (after our experience with logging)
2. I think the issue is a strange combination of list() calls that consume iterators. The retry probably needs to be re-written? (It may predate the rewrite that resulted in the google.cloud.iterator module.)

[dhermes]

• https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/2804

[liyanhui1228]

It might because we cannot read the metric right after it is available to read, and there is a long and unstable cool down period. I think we can have a try to increase the waiting time between each retry by increasing the backoff, and add a RetryErrors decorator to this test function to run again if failed.