Tuning AWS Rust SDK HTTP request settings for latency-aware Amazon DynamoDB applications

[ymwjbxxq]

Describe the bug

As per discussion:
#557

The DynamoDB SDK does not retry based on the with_max_attempts, but its exits at the first failure.

The workaround is to put the queries inside a custom code:

let mut retry_count = 1;
while retry_count < 5 {
     let result = RunQuery().await;
     if let Ok(output) = result {
                //do something or break
      } else {
          retry_count += 1;
      }
}

Expected Behavior

If I set a timeout at 30ms and 5 retries, for example

I expect the Lambda Duration to be between > 30ms and 150ms based on retries and not exiting immediately.

Current Behavior

The SDK does not retry at all. I think it is for all the clients and not just DynamoDB

Reproduction Steps

set the following configuration:

let timeout_config = timeout::Config::new()
        .with_api_timeouts(
            timeout::Api::new()
                // This timeout acts at the "Request to a service" level. When the SDK makes a request to a
                // service, that "request" can contain several HTTP requests. This way, you can retry
                // failures that are likely spurious, or refresh credentials.
                .with_call_timeout(TriState::Set(Duration::from_millis(30)))
                // This timeout acts at the "HTTP request" level and sets a separate timeout for each
                // HTTP request made as part of a "service request."
                .with_call_attempt_timeout(TriState::Set(Duration::from_millis(30))),
        )
        .with_http_timeouts(
            timeout::Http::new()
                // A limit on the amount of time an application takes to attempt to read the first byte over
                // an established, open connection after a write request.
                // Also known as the "time to first byte" timeout.
                .with_read_timeout(TriState::Set(Duration::from_millis(10)))
                // A time limit for completing the connect-handshake. The time starts when
                // making an initial connect attempt on a socket.
                .with_connect_timeout(TriState::Set(Duration::from_millis(10))),
        );
    let config = aws_config::from_env()
        .retry_config(RetryConfig::new().with_max_attempts(10))
        .timeout_config(timeout_config)
        .load()
        .await;

    let dynamodb_client = aws_sdk_dynamodb::Client::new(&config);

you will get this message:
request has timed out: API call (single attempt) timeout occurred after 30ms.

But you will notice that the Lambda Duration is 32ms, so the SDK did not retry.

Possible Solution

Implement retries with exponential backoff like the other AWS-SDK

Additional Information/Context

Libs:

aws-sdk-dynamodb = "0.13.0"
lambda_http = "0.5.0"
lambda_runtime = "0.5.1"
rustc 1.61.0 

Version

cargo tree | grep aws- ├── aws-config v0.13.0 │   ├── aws-http v0.13.0 │   │   ├── aws-smithy-http v0.43.0 │   │   │   ├── aws-smithy-types v0.43.0 │   │   ├── aws-smithy-types v0.43.0 (*) │   │   ├── aws-types v0.13.0 │   │   │   ├── aws-smithy-async v0.43.0 │   │   │   ├── aws-smithy-client v0.43.0 │   │   │   │   ├── aws-smithy-async v0.43.0 (*) │   │   │   │   ├── aws-smithy-http v0.43.0 (*) │   │   │   │   ├── aws-smithy-http-tower v0.43.0 │   │   │   │   │   ├── aws-smithy-http v0.43.0 (*) │   │   │   │   ├── aws-smithy-protocol-test v0.43.0 │   │   │   │   ├── aws-smithy-types v0.43.0 (*) │   │   │   ├── aws-smithy-http v0.43.0 (*) │   │   │   ├── aws-smithy-types v0.43.0 (*) │   ├── aws-sdk-sso v0.13.0 │   │   ├── aws-endpoint v0.13.0 │   │   │   ├── aws-smithy-http v0.43.0 (*) │   │   │   ├── aws-types v0.13.0 (*) │   │   ├── aws-http v0.13.0 (*) │   │   ├── aws-sig-auth v0.13.0 │   │   │   ├── aws-sigv4 v0.13.0 │   │   │   │   ├── aws-smithy-http v0.43.0 (*) │   │   │   ├── aws-smithy-http v0.43.0 (*) │   │   │   ├── aws-types v0.13.0 (*) │   │   ├── aws-smithy-async v0.43.0 (*) │   │   ├── aws-smithy-client v0.43.0 (*) │   │   ├── aws-smithy-http v0.43.0 (*) │   │   ├── aws-smithy-http-tower v0.43.0 (*) │   │   ├── aws-smithy-json v0.43.0 │   │   │   └── aws-smithy-types v0.43.0 (*) │   │   ├── aws-smithy-types v0.43.0 (*) │   │   ├── aws-types v0.13.0 (*) │   ├── aws-sdk-sts v0.13.0 │   │   ├── aws-endpoint v0.13.0 (*) │   │   ├── aws-http v0.13.0 (*) │   │   ├── aws-sig-auth v0.13.0 (*) │   │   ├── aws-smithy-async v0.43.0 (*) │   │   ├── aws-smithy-client v0.43.0 (*) │   │   ├── aws-smithy-http v0.43.0 (*) │   │   ├── aws-smithy-http-tower v0.43.0 (*) │   │   ├── aws-smithy-query v0.43.0 │   │   │   ├── aws-smithy-types v0.43.0 (*) │   │   ├── aws-smithy-types v0.43.0 (*) │   │   ├── aws-smithy-xml v0.43.0 │   │   ├── aws-types v0.13.0 (*) │   ├── aws-smithy-async v0.43.0 (*) │   ├── aws-smithy-client v0.43.0 (*) │   ├── aws-smithy-http v0.43.0 (*) │   ├── aws-smithy-http-tower v0.43.0 (*) │   ├── aws-smithy-json v0.43.0 (*) │   ├── aws-smithy-types v0.43.0 (*) │   ├── aws-types v0.13.0 (*) ├── aws-sdk-cloudfront v0.13.0 │   ├── aws-endpoint v0.13.0 (*) │   ├── aws-http v0.13.0 (*) │   ├── aws-sig-auth v0.13.0 (*) │   ├── aws-smithy-async v0.43.0 (*) │   ├── aws-smithy-client v0.43.0 (*) │   ├── aws-smithy-http v0.43.0 (*) │   ├── aws-smithy-http-tower v0.43.0 (*) │   ├── aws-smithy-types v0.43.0 (*) │   ├── aws-smithy-xml v0.43.0 (*) │   ├── aws-types v0.13.0 (*) ├── aws-sdk-dynamodb v0.13.0 │   ├── aws-endpoint v0.13.0 (*) │   ├── aws-http v0.13.0 (*) │   ├── aws-sig-auth v0.13.0 (*) │   ├── aws-smithy-async v0.43.0 (*) │   ├── aws-smithy-client v0.43.0 (*) │   ├── aws-smithy-http v0.43.0 (*) │   ├── aws-smithy-http-tower v0.43.0 (*) │   ├── aws-smithy-json v0.43.0 (*) │   ├── aws-smithy-types v0.43.0 (*) │   ├── aws-types v0.13.0 (*) ├── aws-sdk-sqs v0.13.0 │   ├── aws-endpoint v0.13.0 (*) │   ├── aws-http v0.13.0 (*) │   ├── aws-sig-auth v0.13.0 (*) │   ├── aws-smithy-async v0.43.0 (*) │   ├── aws-smithy-client v0.43.0 (*) │   ├── aws-smithy-http v0.43.0 (*) │   ├── aws-smithy-http-tower v0.43.0 (*) │   ├── aws-smithy-query v0.43.0 (*) │   ├── aws-smithy-types v0.43.0 (*) │   ├── aws-smithy-xml v0.43.0 (*) │   ├── aws-types v0.13.0 (*) ├── aws-smithy-http v0.43.0 (*) ├── aws-smithy-types v0.43.0 (*) ├── aws-types v0.13.0 (*) ├── aws-smithy-client v0.43.0 (*)

Environment details (OS name and version, etc.)

Mac

Logs

ERROR lambda_runtime: SdkError("request has timed out: API call (single attempt) timeout occurred after 30ms")

[rcoh]

interesting! what operation are you running and what error are you seeing? The SDK doesn't consider every error to be retryable, it's possible that that is the issue? Seeing more logs would also be helpful, could you include the output of aws_smithy_client=trace?

[ymwjbxxq]

interesting! what operation are you running and what error are you seeing? The SDK doesn't consider every error to be retryable, it's possible that that is the issue? Seeing more logs would also be helpful, could you include the output of aws_smithy_client=trace?

I am using a GetItem and GetBatchItem.

Where should I put aws_smithy_client=trace? It is a Lambda :)

I intercept the DynamoDB error in this way:

impl<E> From<SdkError<E>> for ApplicationError
where
    E: error::Error,
{
    fn from(value: SdkError<E>) -> ApplicationError {
        ApplicationError::SdkError(format!("{}", value))
    }
}

So the message that is coming out is:
"request has timed out: API call (single attempt) timeout occurred after XXms"

[rcoh]

ah I see—timeouts are not currently considered retryable. Let me follow up and see if that's considered correct or not. Generally we are very hesitant to retry something that could introduce widespread service degradation by increasing the load.

[ymwjbxxq]

ah, I see—timeouts are not currently considered retryable. Let me follow up and see if that's considered correct or not. Generally, we are very hesitant to retry something that could introduce widespread service degradation by increasing the load.

Other SDKs do it like GO, Node I think all supported retries. Could you tell me what kind of retries are supported?

[rcoh]

The SDK will retry if DynamoDB, e.g. responds with something like ThrottlingError–the difference here is that you aren't getting an error from dynamo, you're making a client side timeout and immediately trying again

[ymwjbxxq]

The best practice from AWS is to set a lower timeout to force the SDK to retry to avoid degradation in performance that you are getting out of the box using the default timeout.

It is one of the principal for getting better performance.

Many SDK for other runtime have all default timeouts and retries.
The SDK timeout is often a 2minutes, which is a degradation of performance. AWS in fact advice to set a lower timeout and let the SDK retry

Node:
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Config.html
Golang:
https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/retries-timeouts/

I am sure this retry works out of the box. I used it all the time :)

I am expecting that timeout is also included because it is pretty logical...AWS network has variable latency, and I know that DynamoDB should respond under the 20ms, so instead of waiting 300ms, I force the timeout for the HTTP request that SDK does to get a quick response.

I think this will not bring any surprise to anyone, but maybe if you guys are afraid maybe, you can add new:
.retry_config specifically for Timeout case or something to include it.

Currently, you do not timeout any call. Instead, you just let it run until maybe the Lambda is timing out (this will hide the real problem), and so this is worse than not having anything.
For example, as default, you can setup 2 minutes and 3 retries like the other SDK, and as a user, I can setup my client.

In node, for example, is like this:

const options: ServiceConfigurationOptions = {
  region: process.env.AWS_REGION,
  maxRetries: 10,
  httpOptions: {
    timeout: 100, //The number of milliseconds a request can take before automatically being terminated. Defaults to two minutes (120000).
  },
};

[rcoh]

you're right—I think this is exactly how the Java SDK behaves today as well. This is definitely a bug, and one I hope to get a fix out for shortly