Need to optimize HTTP client stack for S3 #1350
russfellows
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
There are currently performance limitations with the default Rust SDK for S3, which includes a WIDE variety of crates including, but not limited to the AWS S3 SDK, AWS Smithy HTTP Client, Hyper, Rustls, and many others. This is documented in issue #1349 .
In order for S3 to be useful for HPC workloads, and in particular AI, it is critical that the performance levels approach that of NFS or other networked storage protocols, and even NVMe-oF. Currently, this is not the case, due to several inefficiencies in the client software stack.
On this note, RDMA technologies should also be leveraged and available for S3 access, in order to fully maximize the network and other resources.
Beta Was this translation helpful? Give feedback.
All reactions