Support Reusable Request Bodies and Improve Payload Handling

[bdraco]

Overview

This PR enhances aiohttp's payload handling system to support reusable request bodies, addressing a common pain point where payloads could only be used once. The changes enable retry mechanisms and middleware to work with request bodies without consuming them permanently.

Fixes #5577 - Digest authentication now works with request bodies (previously failed with "payload consumed" errors)
Fixes #5530 - Request bodies can now be reused in retry scenarios without "attempt to read body twice" errors
Replaces and closes #9201 - Thanks @GLGDLY

Key Changes

1. Reusable Payload Support

• Added consumed and autoclose properties to track payload state
• Implemented as_bytes() method for safe payload content retrieval
• Added caching mechanism for AsyncIterablePayload to enable reuse
• File-based payloads now remember their start position for re-reading

2. Enhanced Payload Classes

• BytesPayload: Marked as autoclose=True since bytes are immutable and reusable
• BytesIOPayload: Optimized size property using cached_property
• IOBasePayload: Added position tracking and restoration for file handles
• AsyncIterablePayload: Implemented chunk caching to allow multiple reads

3. ClientRequest Improvements

• Added update_body() method for middleware use to safely update request bodies
• Made sure the .body setter cleans up all current payload types, and documented that using the setter is the wrong way to do this as we want consumers of the API to call the async update_body() method so we can safely clean up any type of new payloads in the future. A resource warning will be thrown if they manually set the body using the setter instead of calling update_body(), as this is likely to leak in the future.
• Improved handling of empty bodies and proper cleanup
• Better separation between body preparation and sending

4. Middleware Enhancements

• Updated digest authentication middleware to use new reusable payload features
• Added support for middleware to safely inspect and modify request bodies
• Improved error handling for consumed payloads

5. Resource Management

• Implemented proper close() methods for all payload types
• Fixed resource leaks by scheduling file closes without awaiting
• Better handling of payload cleanup during redirects and errors

Benefits

• Enables retry middleware to resend requests with bodies
• Allows digest authentication to work with request bodies
• Supports middleware that needs to inspect request content
• Prevents "payload already consumed" errors in common scenarios
• Improves resource cleanup and prevents file descriptor leaks

Testing

• Added comprehensive tests for payload reusability
• Tests for middleware interaction with request bodies
• Benchmarks to ensure no performance regression
• Edge case testing for file position restoration and async iterables

Breaking Changes

None - all changes are backward compatible. Existing code will continue to work as before.

Documentation

• Updated client reference documentation
• Added examples of using reusable payloads
• Documented best practices for middleware authors

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.81%. Comparing base (4824648) to head (7be3123).
⚠️ Report is 378 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #11017      +/-   ##
==========================================
+ Coverage   98.75%   98.81%   +0.06%     
==========================================
  Files         129      129              
  Lines       40080    41308    +1228     
  Branches     2204     2226      +22     
==========================================
+ Hits        39580    40819    +1239     
+ Misses        348      339       -9     
+ Partials      152      150       -2     

Flag	Coverage Δ
CI-GHA	98.70% <100.00%> (+0.06%)	⬆️
OS-Linux	98.42% <100.00%> (+0.11%)	⬆️
OS-Windows	96.67% <100.00%> (+0.10%)	⬆️
OS-macOS	97.61% <100.00%> (+0.07%)	⬆️
Py-3.10.11	97.50% <100.00%> (+0.06%)	⬆️
Py-3.10.17	98.01% <100.00%> (+0.67%)	⬆️
Py-3.11.12	98.09% <99.84%> (+0.23%)	⬆️
Py-3.11.9	97.57% <99.84%> (+0.06%)	⬆️
Py-3.12.10	98.47% <100.00%> (+0.85%)	⬆️
Py-3.13.3	98.46% <100.00%> (+1.13%)	⬆️
Py-3.9.13	97.38% <99.92%> (+0.06%)	⬆️
Py-3.9.22	97.88% <99.92%> (+0.68%)	⬆️
Py-pypy7.3.16	85.42% <99.62%> (?)
VM-macos	97.61% <100.00%> (+0.07%)	⬆️
VM-ubuntu	98.42% <100.00%> (+0.11%)	⬆️
VM-windows	96.67% <100.00%> (+0.10%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[bdraco]

I'm going to try to get this merged tonight so its not blocking Sam's PRs. I keep finding small nits though

[patchback]

Backport to 3.12: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 1c01726 on top of patchback/backports/3.12/1c0172666f26e11e99850198154336390fdacb09/pr-11017

Backporting merged PR #11017 into master

1. Ensure you have a local repo clone of your fork. Unless you cloned it
   from the upstream, this would be your origin remote.
2. Make sure you have an upstream repo added as a remote too. In these
   instructions you'll refer to it by the name upstream. If you don't
   have it, here's how you can add it:

   $ git remote add upstream https://github.com/aio-libs/aiohttp.git

3. Ensure you have the latest copy of upstream and prepare a branch
   that will hold the backported code:

   $ git fetch upstream
   $ git checkout -b patchback/backports/3.12/1c0172666f26e11e99850198154336390fdacb09/pr-11017 upstream/3.12

4. Now, cherry-pick PR Support Reusable Request Bodies and Improve Payload Handling #11017 contents into that branch:

   $ git cherry-pick -x 1c0172666f26e11e99850198154336390fdacb09

   If it'll yell at you with something like fatal: Commit 1c0172666f26e11e99850198154336390fdacb09 is a merge but no -m option was given., add -m 1 as follows instead:

   $ git cherry-pick -m1 -x 1c0172666f26e11e99850198154336390fdacb09

5. At this point, you'll probably encounter some merge conflicts. You must
   resolve them in to preserve the patch from PR Support Reusable Request Bodies and Improve Payload Handling #11017 as close to the
   original as possible.
6. Push this branch to your fork on GitHub:

   $ git push origin patchback/backports/3.12/1c0172666f26e11e99850198154336390fdacb09/pr-11017

7. Create a PR, ensure that the CI is green. If it's not — update it so that
   the tests and any other checks pass. This is it!
   Now relax and wait for the maintainers to process your pull request
   when they have some cycles to do reviews. Don't worry — they'll tell you if
   any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

[patchback]

Backport to 3.13: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 1c01726 on top of patchback/backports/3.13/1c0172666f26e11e99850198154336390fdacb09/pr-11017

Backporting merged PR #11017 into master

1. Ensure you have a local repo clone of your fork. Unless you cloned it
   from the upstream, this would be your origin remote.
2. Make sure you have an upstream repo added as a remote too. In these
   instructions you'll refer to it by the name upstream. If you don't
   have it, here's how you can add it:

   $ git remote add upstream https://github.com/aio-libs/aiohttp.git

3. Ensure you have the latest copy of upstream and prepare a branch
   that will hold the backported code:

   $ git fetch upstream
   $ git checkout -b patchback/backports/3.13/1c0172666f26e11e99850198154336390fdacb09/pr-11017 upstream/3.13

4. Now, cherry-pick PR Support Reusable Request Bodies and Improve Payload Handling #11017 contents into that branch:

   $ git cherry-pick -x 1c0172666f26e11e99850198154336390fdacb09

   If it'll yell at you with something like fatal: Commit 1c0172666f26e11e99850198154336390fdacb09 is a merge but no -m option was given., add -m 1 as follows instead:

   $ git cherry-pick -m1 -x 1c0172666f26e11e99850198154336390fdacb09

5. At this point, you'll probably encounter some merge conflicts. You must
   resolve them in to preserve the patch from PR Support Reusable Request Bodies and Improve Payload Handling #11017 as close to the
   original as possible.
6. Push this branch to your fork on GitHub:

   $ git push origin patchback/backports/3.13/1c0172666f26e11e99850198154336390fdacb09/pr-11017

7. Create a PR, ensure that the CI is green. If it's not — update it so that
   the tests and any other checks pass. This is it!
   Now relax and wait for the maintainers to process your pull request
   when they have some cycles to do reviews. Don't worry — they'll tell you if
   any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.