Support dynamically updating authentication headers

[faph]

Is your feature request related to a problem?

See #6908 for discussion with @Dreamsorcerer

We are have a continuously running service that makes HTTP requests using aiohttp's client. These requests are sent with a bearer token in the authorization header. This token needs to be refreshed from time to time.

Therefore we cannot set the header on the client object. We need to evaluate the header prior to making individual requests. This is cumbersome for an app developer.

Describe the solution you'd like

We would like to configure the client with the required data which can be used at each request to populate an authorization header. In its simplest form, this could be a simple co-routine that evaluates a token cache, generates a new token if required, and populate the header as required.

For the avoidance of doubt, the proposal is for wherever aiohttp supports an auth argument (client session, request), it would support the "enhanced" auth handler.

Currently, aiohttp supports a BasicAuth object which is a customized namedtuple (fields: username, password) with an encode method which returns the base64 encoded header value. Runtime checks are performed ensuring the auth value is an instance of that class.

TODO: expand on preferred optionSee options in comment below.

Describe alternatives you've considered

Both the requests library and the httpx library have similar functionality. For reference:

• https://requests.readthedocs.io/en/latest/user/authentication/#new-forms-of-authentication
• https://www.python-httpx.org/advanced/#customizing-authentication

The httpx approach supports a more defined base class/interface for an auth handler, whereas requests just requires a callable. Fundamentally they do the same thing and inject an authorization header in the request headers.

Related component

Client

Additional context

I am probably able to contribute a PR with an implementation, depending on the agreed scope of the changes.

Code of Conduct

• I agree to follow the aio-libs Code of Conduct

[faph]

Implementation options

1. Do nothing. App developers can pass in headers manually as and when required. Or alternatively they could subclass ClientSession or ClientRequest (ClientSession is marked as "final" and should not be subclassed).

2. Add support for a generic coroutine function to be passed in for the auth argument. This would be in addition to the currently supported BasicAuth class. The coroutine function would accept a single argument, the ClientRequest instance and the coroutine would be free to modify the request (with the intention to just modify headers...). ClientRequest.send would call/await this coroutine.

3. Similar to option 2, but a single AuthHandler base class would be implemented with an empty async __call__ method. A __call__ method could be added to existing BasicAuth class which simply delegates to the existing encode synchronous method. [I guess the base class is really optional here since it’s-just-something-that-returns-an-awaitable]

4. Tweak to options 2 and 3 above: the coroutine function would not accept any arguments and should just return a value for an Authorization header (or nothing). In that case, we could also accept a coroutine object instead.

5. Add support for more generic "pre-send" and "post-receive" hooks (both on client instance, and individual request methods). These hooks would be free to modify anything on the request and the response respectively.

[faph]

My personal preference would be towards option 3. It’s simple, clean and would feel very familiar for people coming from “requests” lib.

[faph]

@Dreamsorcerer and maintainers, I welcome feedback on this.

[faph]

I created draft PR #6954 just to test out how the existing BasicAuth functionality could be generalized.

[faph]

Just picking up some feedback from @webknjaz here: #6954 (comment) (thanks!)

Personally, I don't feel adding an auth handler base class adds that much if the contract really is that it must be a co-routine function and just that. I'm not sure if that is related to the comment regarding the protocol...

Libraries like requests and httpx do both, they allow plain functions but also provide a base class to inherit from if one wishes. In case of httpx that does makes sense since it supports both async and async logic. But if it's a class with just a single method, you may as well not bother. Unless you want to give a hint to app developers that they can provide a "rich object" with its own state instead of a stateless co-routine.

[faph]

For illustration, the expected usage is something like this:

async def my_auth(request: ClientRequest) -> None:
    token = await token_from_cache()
    if not token:
        token = await generate_token()
    request.headers["Authorization"] = f"Bearer {token}"

async with ClientSession(auth=my_auth) as session:
    url = "..."
    async with session.get(url) as resp:
        print(resp.status)

# OR

async with ClientSession() as session:
    url = "..."
    async with session.get(url, auth=my_auth) as resp:
        print(resp.status)

As a class-based solution, it could look like this:

class MyAuth:
    def __init__(self, identity_provider):
        self._identity_provider = identity_provider
        self._token_cache = ...

    async def __call__(self, request: ClientRequest) -> None:
        token = await self._token_cache.get()
        if not token:
            token = await generate_token(self._identity_provider)
            await self._token_cache.set(token)
        request.headers["Authorization"] = f"Bearer {token}"

my_identity_provider = ...

async with ClientSession(auth=MyAuth(my_identity_provider) as session:
    url = "..."
    async with session.get(url) as resp:
        print(resp.status)

[Dreamsorcerer]

Personally, I don't feel adding an auth handler base class adds that much if the contract really is that it must be a co-routine function and just that.

My feeling is that we should probably just accept a coroutine and keep it generic. You can still pass a method from a more complex object, if you want to store state or something.

I also think that calling this auth would be a mistake, even if that's what other libraries are doing. The API doesn't seem to do anything specific to auth, I'd say this is a generic middleware, not dissimilar to the middlewares we support on the server side.

So, for the existing BasicAuth, you could end up with an approach like:

auth = BasicAuth("user")
async with ClientSession(middleware=auth.middleware):
    ...

With that new method being defined something like: https://github.com/aio-libs/aiohttp/pull/6954/files#diff-b797dd8733928df191ba2061121ab8b69976c185fcbfad4534891d3252b9ac30R165

---

async def my_auth(request: ClientRequest) -> None:
    token = await token_from_cache()
    if not token:
        token = await generate_token()
    request.headers["Authorization"] = f"Bearer {token}"

async with ClientSession(auth=my_auth) as session:
    ...

In this example though, I'm wondering why we wouldn't have access to the ClientSession in our middleware. Would it be useful to access the session as well? Would it be an improvement if this code only updated session.headers each time the token expires?

---

Though I still think this would be more efficient without something being called on every request. How does token_from_cache() know that the token has expired? If it's just time based, it seems to me that it would make more sense to just setup a callback, something like this (which should work without any changes):

async def set_token(session, event):
    while True:
        async with session.post("get_token") as resp:
            token = await resp.json()
            session.headers["Authorization"] = token["auth"]
            event.set()
            await asyncio.sleep(token["valid_duration"])

async with ClientSession() as session:
    ready = asyncio.Event()
    t = asyncio.create_task(set_token(session, event))
    await ready.wait()
    ...   # Your application code here...
    t.cancel()
    await t

Maybe to make that safer, there could be a Lock passed to ClientSession that is acquired during the auth call and blocks any requests until the new token is set. However, in most cases you can probably get away with just using valid_duration - 10 or similar, to ensure the request happens before the token gets expired.

[Dreamsorcerer]

@webknjaz Any thoughts on this?

[faph]

@Dreamsorcerer thanks! Let me separate this out into a few distinct decision points below

[faph]

Auth-specific vs generic middleware/hook

This is really options 2-4 vs option 5 in #6915 (comment). I don't have a strong preference for either approach.

Some advantages for doing something auth-specific:

1. There is an existing auth argument.
2. It's recognizable for app devs coming from other HTTP client libs.

Generic advantages:

1. More flexible.
2. Avoids a generic solution, mislabeled as auth-specific.

The only thing with the generic solution is that there is some proxy-related auth logic in the client, so we need to make sure that this interacts as expected.

[faph]

Client session property vs request property

I agree that in many case, one would set an auth handler on the session. However, the auth handler is in reality a property of an individual request, not the session. Different requests within a single session may require different authentication (or no authentication).

The existing logic is correct in this sense whereby the session can provide a default auth value for use by any request. Similarly, the implicit session API (aiohttp.request()) should equally allow auth to be set.

[faph]

Background token refresh vs on-send function call

In a typical token expiring case, one could rotate the auth header using a background task. But I think it would involve setting ClientSession._default_auth or ClientSession._default_headers. I guess making this publicly settable could be considered.

However, this would only work for default auth handlers on a per-session basis. Not for individual requests.

I don't understand the concern about an on-the-fly function call to evaluate/populate the request's headers. Each ClientRequest object instantiation already goes through a series of functional calls. The only real change here that there's a pluggable hook which accepts a co-routine function. Obviously, any token cache fetch logic should be as fast as it can be...