S3 object XML parsing error for large files #209
Description
Bug Report
Version
├── aws-sdk-s3 v0.0.16-alpha (https://github.com/awslabs/aws-sdk-rust?tag=v0.0.16-alpha#dab3b0a9)
│ ├── aws-auth v0.1.0 (https://github.com/awslabs/aws-sdk-rust?tag=v0.0.16-alpha#dab3b0a9) (*)
│ ├── aws-endpoint v0.1.0 (https://github.com/awslabs/aws-sdk-rust?tag=v0.0.16-alpha#dab3b0a9) (*)
│ ├── aws-http v0.1.0 (https://github.com/awslabs/aws-sdk-rust?tag=v0.0.16-alpha#dab3b0a9) (*)
│ ├── aws-hyper v0.1.0 (https://github.com/awslabs/aws-sdk-rust?tag=v0.0.16-alpha#dab3b0a9) (*)
│ ├── aws-sig-auth v0.1.0 (https://github.com/awslabs/aws-sdk-rust?tag=v0.0.16-alpha#dab3b0a9) (*)
│ ├── aws-types v0.1.0 (https://github.com/awslabs/aws-sdk-rust?tag=v0.0.16-alpha#dab3b0a9) (*)
│ ├── smithy-http v0.0.1 (https://github.com/awslabs/aws-sdk-rust?tag=v0.0.16-alpha#dab3b0a9) (*)
│ ├── smithy-types v0.0.1 (https://github.com/awslabs/aws-sdk-rust?tag=v0.0.16-alpha#dab3b0a9) (*)
│ └── smithy-xml v0.1.0 (https://github.com/awslabs/aws-sdk-rust?tag=v0.0.16-alpha#dab3b0a9)
Platform
Linux DELL-XPS-13-9380 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
AWS Services
S3
Description
The list_objects client function produces a XML parsing error for large files because the model::Object has a size (in bytes) with type i32, which I believe maxes out around 2GB.
https://github.com/awslabs/aws-sdk-rust/blob/main/sdk/s3/src/model.rs#L11230-L11231
https://github.com/awslabs/aws-sdk-rust/blob/main/sdk/s3/src/xml_deser.rs#L3958-L3961
Example code:
let client = aws_sdk_s3::Client::from_env();
let result = client
.list_objects()
.bucket(&bucket)
.prefix(&key)
.send()
.await;I expected an Ok result
Instead I got the following ListObjectsError:
ServiceError { err: ListObjectsError { kind: Unhandled(Custom("expected (integer: `com.amazonaws.s3#Size`)")) ...
With the following relevant trace logs:
[2021-08-27T20:27:54Z TRACE smithy_http_tower::dispatch] request=Request { method: GET, uri: REDACTED, version: HTTP/1.1, headers: {"content-type": "application/xml", "content-length": "0", "host": "s3.us-east-1.amazonaws.com", "user-agent": "aws-sdk-rust/0.1.0 os/linux lang/rust/1.53.0", "x-amz-user-agent": "aws-sdk-rust/0.1.0 api/s3/0.0.16-alpha os/linux lang/rust/1.53.0", "authorization": Sensitive, "x-amz-date": "20210827T202754Z", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}, body: SdkBody { inner: Once(Some(b"")), retryable: true } }
[2021-08-27T20:27:54Z TRACE smithy_http::middleware] http_response=Response { status: 200, version: HTTP/1.1, headers: {"x-amz-id-2": "jXU2ZMxiZIVhIp0Dq+xJ82p+vnsHmTwdFFdAGjexwnghDvmkx7QWeM4+qn6opKZxM3Vkf2aG010=", "x-amz-request-id": "HDC393J1HSBX3PVY", "date": "Fri, 27 Aug 2021 20:27:55 GMT", "x-amz-bucket-region": "us-east-1", "content-type": "application/xml", "transfer-encoding": "chunked", "server": "AmazonS3"}, body: b"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<ListBucketResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Name>REDACTED</Name><Prefix>REDACTED</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>REDACTED</Key><LastModified>2021-08-26T23:00:01.000Z</LastModified><ETag>"fd71cdbe67710902513479d9650b0785"</ETag><Size>2940645877</Size><Owner><ID>REDACTED</ID><DisplayName>REDACTED</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>" }
Notice the part <Size>2940645877</Size>, which is greater than the max value of i32: 2147483647