Refresh credentials in the background during preempt expiry

[dscpinheiro]

Describe the feature

In V4, we attempted to perform a background refresh of credentials during their preempt expiry period (#3541), however we received several customer reports of expired credentials being used and had to revert the change (#4023).

The background refresh makes sense as a feature request (it was attempting to improve scenarios such as #2464), but we need to fix the proposed implementation before re-introducing the behavior.

[dscpinheiro]

Here are some notes I took before reverting the PR (for reference only, relevant to the .NET SDK team):

When a RefreshingAWSCredentials (which several of our credential providers inherit from) instance is created with a lifetime of 60 minutes and a preempt expiry of 15 minutes, it sets its expiration to 45 minutes. This logic has been in the SDK for quite some time, it existed even in V3: https://github.com/aws/aws-sdk-net/blob/aws-sdk-net-v3.7-development/sdk/src/Core/Amazon.Runtime/Credentials/RefreshingAWSCredentials.cs#L193-L195

When a GetCredentials call is made before the preempt expire time (for example, at T0 + 15 minutes) the credentials are valid so returned right away. If the call is made after the current expiration (so T0 + 46 minutes or T0 + 61 minutes), the credentials are refreshed synchronously before being returned.

Let's say a call is made at T0 + 39 minutes. The credentials are still valid so should be returned, but with the current logic we're in the preempt expire again (because 39 is greater than 45 - 15) and we should refresh them in the background. This is not happening because the IsExpiredWithin method doesn't take the preempt expire time into account, it just checks if now > expiration.

I tried the suggestion mentioned in one of other issues (to change the calculation in the GetTimeToLive) but that breaks some other existing tests related to static stability.

[robin-karlsson]

Just adding a comment here with reference to the closed ticket #4053 . Having the semaphore check this early when trying to get credentials is stalling our service from sending events to SQS. We're running the service in EKS ending up with a credentials derived from RefreshingAWSCredentials.