Skip to content

Aws creds error #2380

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Aws creds error #2380

wants to merge 2 commits into from

Conversation

@revmischa
Copy link
Contributor

This PR contains:

  • New features
  • Changes to dev-tools e.g. CI config / github tooling
  • Docs
  • Bug fixes
  • Code refactor

What is the current behavior? (You can also link to an open issue here)

If AWS creds are expired it prints:

[08/31/25 17:49:10] WARNING  SSO token refresh attempt failed                                                                                                                                                                                                                                                                                                                                                                         tokens.py:128
                             Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                                                                    
                               File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 126, in _refresh_access_token                                                                                                                                                                                                                                                                                                       
                                 return await self._attempt_create_token(token)                                                                                                                                                                                                                                                                                                                                                                    
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                    
                               File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 85, in _attempt_create_token                                                                                                                                                                                                                                                                                                        
                                 response = await client.create_token(                                                                                                                                                                                                                                                                                                                                                                             
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                             
                                 ...<4 lines>...                                                                                                                                                                                                                                                                                                                                                                                                   
                                 )                                                                                                                                                                                                                                                                                                                                                                                                                 
                                 ^                                                                                                                                                                                                                                                                                                                                                                                                                 
                               File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/client.py", line 412, in _make_api_call                                                                                                                                                                                                                                                                                                              
                                 raise error_class(parsed_response, operation_name)                                                                                                                                                                                                                                                                                                                                                                
                             botocore.errorfactory.InvalidGrantException: An error occurred (InvalidGrantException) when calling the CreateToken operation:                                                                                                                                                                                                                                                                                        
                    WARNING  Refreshing temporary credentials failed during mandatory refresh period.                                                                                                                                                                                                                                                                                                                            credentials.py:375
                             Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                                                                    
                               File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/credentials.py", line 372, in _protected_refresh                                                                                                                                                                                                                                                                                                     
                                 metadata = await resolve_awaitable(self._refresh_using())                                                                                                                                                                                                                                                                                                                                                         
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                         
                               File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/_helpers.py", line 6, in resolve_awaitable                                                                                                                                                                                                                                                                                                           
                                 return await obj                                                                                                                                                                                                                                                                                                                                                                                                  
                                        ^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                  
                               File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/credentials.py", line 428, in fetch_credentials                                                                                                                                                                                                                                                                                                      
                                 return await self._get_cached_credentials()                                                                                                                                                                                                                                                                                                                                                                       
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                       
                               File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/credentials.py", line 438, in _get_cached_credentials                                                                                                                                                                                                                                                                                                
                                 response = await self._get_credentials()                                                                                                                                                                                                                                                                                                                                                                          
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                          
                               File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/credentials.py", line 1049, in _get_credentials                                                                                                                                                                                                                                                                                                      
                                 token = (await initial_token_data.get_frozen_token()).token                                                                                                                                                                                                                                                                                                                                                       
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                              
                               File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 38, in get_frozen_token                                                                                                                                                                                                                                                                                                             
                                 await self._refresh()                                                                                                                                                                                                                                                                                                                                                                                             
                               File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 51, in _refresh                                                                                                                                                                                                                                                                                                                     
                                 await self._protected_refresh()                                                                                                                                                                                                                                                                                                                                                                                   
                               File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 76, in _protected_refresh                                                                                                                                                                                                                                                                                                           
                                 raise TokenRetrievalError(                                                                                                                                                                                                                                                                                                                                                                                        
                                 ...<2 lines>...                                                                                                                                                                                                                                                                                                                                                                                                   
                                 )                                                                                                                                                                                                                                                                                                                                                                                                                 
                             botocore.exceptions.TokenRetrievalError: Error when retrieving token from sso: Token has expired and refresh failed                                                                                                                                                                                                                                                                                                   
                    WARNING  Refreshing token failed during the mandatory refresh period.                                                                                                                                                                                                                                                                                                                                              tokens.py:65
                             Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                                                                    
                               File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 63, in _protected_refresh                                                                                                                                                                                                                                                                                                           
                                 self._frozen_token = await self._refresh_using()                                                                                                                                                                                                                                                                                                                                                                  
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                  
                               File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 141, in _refresher                                                                                                                                                                                                                                                                                                                  
                                 new_token_dict = await self._refresh_access_token(token_dict)                                                                                                                                                                                                                                                                                                                                                     
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                     
                               File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 126, in _refresh_access_token                                                                                                                                                                                                                                                                                                       
                                 return await self._attempt_create_token(token)                                                                                                                                                                                                                                                                                                                                                                    
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                    
                               File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 84, in _attempt_create_token                                                                                                                                                                                                                                                                                                        
                                 async with self._client as client:                                                                                                                                                                                                                                                                                                                                                                                
                                            ^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                           
                               File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/session.py", line 25, in __aenter__                                                                                                                                                                                                                                                                                                                  
                                 self._client = await self._coro                                                                                                                                                                                                                                                                                                                                                                                   
                                                ^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                   
                             RuntimeError: cannot reuse already awaited coroutine                                                                                                                                                                                                                                                                                                                                                                  

......
snip
.....

    |            ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/click/core.py", line 788, in invoke
    |     return __callback(*args, **kwargs)
    |   File "/home/inspect/app/src/inspect_ai/_cli/common.py", line 45, in wrapper
    |     return cast(click.Context, func(*args, **kwargs))
    |                                ~~~~^^^^^^^^^^^^^^^^^
    |   File "/home/inspect/app/src/inspect_ai/_cli/common.py", line 110, in wrapper
    |     return cast(click.Context, func(*args, **kwargs))
    |                                ~~~~^^^^^^^^^^^^^^^^^
    |   File "/home/inspect/app/src/inspect_ai/_cli/eval.py", line 529, in wrapper
    |     return cast(click.Context, func(*args, **kwargs))
    |                                ~~~~^^^^^^^^^^^^^^^^^
    |   File "/home/inspect/app/src/inspect_ai/_cli/eval.py", line 619, in eval_command
    |     eval_exec(
    |     ~~~~~~~~~^
    |         tasks=tasks,
    |         ^^^^^^^^^^^^
    |     ...<47 lines>...
    |         **config,
    |         ^^^^^^^^^
    |     )
    |     ^
    |   File "/home/inspect/app/src/inspect_ai/_cli/eval.py", line 1052, in eval_exec
    |     eval(**params)
    |     ~~~~^^^^^^^^^^
    |   File "/home/inspect/app/src/inspect_ai/_eval/eval.py", line 272, in eval
    |     return task_display().run_task_app(run_task_app)
    |            ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
    |   File "/home/inspect/app/src/inspect_ai/_display/textual/display.py", line 51, in run_task_app
    |     raise result.value
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/textual/worker.py", line 368, in _run
    |     self._result = await self.run()
    |                    ^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/textual/worker.py", line 352, in run
    |     return await (
    |            ^^^^^^^
    |         self._run_threaded() if self._thread_worker else self._run_async()
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |     )
    |     ^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/textual/worker.py", line 339, in _run_async
    |     return await self._work
    |            ^^^^^^^^^^^^^^^^
    |   File "/home/inspect/app/src/inspect_ai/_eval/eval.py", line 221, in run_task_app
    |     return await eval_async(
    |            ^^^^^^^^^^^^^^^^^
    |     ...<42 lines>...
    |     )
    |     ^
    |   File "/home/inspect/app/src/inspect_ai/_eval/eval.py", line 447, in eval_async
    |     raise inner_exception(ex)
    |   File "/home/inspect/app/src/inspect_ai/_eval/eval.py", line 395, in run
    |     result = await _eval_async_inner(
    |              ^^^^^^^^^^^^^^^^^^^^^^^^
    |     ...<43 lines>...
    |     )
    |     ^
    |   File "/home/inspect/app/src/inspect_ai/_eval/eval.py", line 722, in _eval_async_inner
    |     raise e
    |   File "/home/inspect/app/src/inspect_ai/_eval/eval.py", line 580, in _eval_async_inner
    |     recorder = create_recorder_for_format(log_format or DEFAULT_LOG_FORMAT, log_dir)
    |   File "/home/inspect/app/src/inspect_ai/log/_recorders/create.py", line 14, in create_recorder_for_format
    |     return recorder(*args, **kwargs)
    |   File "/home/inspect/app/src/inspect_ai/log/_recorders/eval.py", line 72, in __init__
    |     super().__init__(log_dir, ".eval", fs_options)
    |     ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/inspect/app/src/inspect_ai/log/_recorders/file.py", line 28, in __init__
    |     self.fs.mkdir(self.log_dir, exist_ok=True)
    |     ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/inspect/app/src/inspect_ai/_util/file.py", line 194, in mkdir
    |     self.fs.makedir(path, create_parents=False)
    |     ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/fsspec/spec.py", line 1727, in makedir
    |     return self.mkdir(path, create_parents=create_parents, **kwargs)
    |            ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/fsspec/asyn.py", line 118, in wrapper
    |     return sync(self.loop, func, *args, **kwargs)
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/fsspec/asyn.py", line 103, in sync
    |     raise return_result
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/fsspec/asyn.py", line 56, in _runner
    |     result[0] = await coro
    |                 ^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/s3fs/core.py", line 963, in _mkdir
    |     await self._ls(bucket)
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/s3fs/core.py", line 1040, in _ls
    |     files = await self._lsdir(path, refresh, versions=versions)
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/s3fs/core.py", line 755, in _lsdir
    |     async for c in self._iterdir(
    |     ...<9 lines>...
    |             files.append(c)
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/s3fs/core.py", line 805, in _iterdir
    |     async for i in it:
    |     ...<14 lines>...
    |                 yield c
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/paginate.py", line 30, in __anext__
    |     response = await self._make_request(current_kwargs)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/client.py", line 394, in _make_api_call
    |     http, parsed_response = await self._make_request(
    |                             ^^^^^^^^^^^^^^^^^^^^^^^^^
    |         operation_model, request_dict, request_context
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |     )
    |     ^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/client.py", line 420, in _make_request
    |     return await self._endpoint.make_request(
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |         operation_model, request_dict
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |     )
    |     ^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/endpoint.py", line 109, in _send_request
    |     request = await self.create_request(request_dict, operation_model)
    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/endpoint.py", line 97, in create_request
    |     await self._event_emitter.emit(
    |     ...<3 lines>...
    |     )
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/hooks.py", line 68, in _emit
    |     response = await resolve_awaitable(handler(**kwargs))
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/_helpers.py", line 6, in resolve_awaitable
    |     return await obj
    |            ^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/signers.py", line 24, in handler
    |     return await self.sign(operation_name, request)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/signers.py", line 81, in sign
    |     auth = await self.get_auth_instance(**kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/signers.py", line 164, in get_auth_instance
    |     frozen_credentials = await credentials.get_frozen_credentials()
    |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/credentials.py", line 403, in get_frozen_credentials
    |     await self._refresh()
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/credentials.py", line 356, in _refresh
    |     await self._protected_refresh(
    |         is_mandatory=is_mandatory_refresh
    |     )
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/credentials.py", line 372, in _protected_refresh
    |     metadata = await resolve_awaitable(self._refresh_using())
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/_helpers.py", line 6, in resolve_awaitable
    |     return await obj
    |            ^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/credentials.py", line 428, in fetch_credentials
    |     return await self._get_cached_credentials()
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/credentials.py", line 438, in _get_cached_credentials
    |     response = await self._get_credentials()
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/credentials.py", line 1049, in _get_credentials
    |     token = (await initial_token_data.get_frozen_token()).token
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 38, in get_frozen_token
    |     await self._refresh()
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 51, in _refresh
    |     await self._protected_refresh()
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 63, in _protected_refresh
    |     self._frozen_token = await self._refresh_using()
    |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 141, in _refresher
    |     new_token_dict = await self._refresh_access_token(token_dict)
    |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 126, in _refresh_access_token
    |     return await self._attempt_create_token(token)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 84, in _attempt_create_token
    |     async with self._client as client:
    |                ^^^^^^^^^^^^
    |   File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/session.py", line 25, in __aenter__
    |     self._client = await self._coro
    |                    ^^^^^^^^^^^^^^^^
    | RuntimeError: cannot reuse already awaited coroutine
    +------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/inspect_ai/bin/inspect", line 7, in <module>
    sys.exit(main())
             ~~~~^^
  File "/home/inspect/app/src/inspect_ai/_cli/main.py", line 56, in main
    inspect(auto_envvar_prefix="INSPECT")  # pylint: disable=no-value-for-parameter
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/opt/inspect_ai/lib/python3.13/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/home/inspect/app/src/inspect_ai/_cli/common.py", line 45, in wrapper
    return cast(click.Context, func(*args, **kwargs))
                               ~~~~^^^^^^^^^^^^^^^^^
  File "/home/inspect/app/src/inspect_ai/_cli/common.py", line 110, in wrapper
    return cast(click.Context, func(*args, **kwargs))
                               ~~~~^^^^^^^^^^^^^^^^^
  File "/home/inspect/app/src/inspect_ai/_cli/eval.py", line 529, in wrapper
    return cast(click.Context, func(*args, **kwargs))
                               ~~~~^^^^^^^^^^^^^^^^^
  File "/home/inspect/app/src/inspect_ai/_cli/eval.py", line 619, in eval_command
    eval_exec(
    ~~~~~~~~~^
        tasks=tasks,
        ^^^^^^^^^^^^
    ...<47 lines>...
        **config,
        ^^^^^^^^^
    )
    ^
  File "/home/inspect/app/src/inspect_ai/_cli/eval.py", line 1052, in eval_exec
    eval(**params)
    ~~~~^^^^^^^^^^
  File "/home/inspect/app/src/inspect_ai/_eval/eval.py", line 272, in eval
    return task_display().run_task_app(run_task_app)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/home/inspect/app/src/inspect_ai/_display/textual/display.py", line 51, in run_task_app
    raise result.value
  File "/opt/inspect_ai/lib/python3.13/site-packages/textual/worker.py", line 368, in _run
    self._result = await self.run()
                   ^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/textual/worker.py", line 352, in run
    return await (
           ^^^^^^^
        self._run_threaded() if self._thread_worker else self._run_async()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/inspect_ai/lib/python3.13/site-packages/textual/worker.py", line 339, in _run_async
    return await self._work
           ^^^^^^^^^^^^^^^^
  File "/home/inspect/app/src/inspect_ai/_eval/eval.py", line 221, in run_task_app
    return await eval_async(
           ^^^^^^^^^^^^^^^^^
    ...<42 lines>...
    )
    ^
  File "/home/inspect/app/src/inspect_ai/_eval/eval.py", line 447, in eval_async
    raise inner_exception(ex)
  File "/home/inspect/app/src/inspect_ai/_eval/eval.py", line 395, in run
    result = await _eval_async_inner(
             ^^^^^^^^^^^^^^^^^^^^^^^^
    ...<43 lines>...
    )
    ^
  File "/home/inspect/app/src/inspect_ai/_eval/eval.py", line 722, in _eval_async_inner
    raise e
  File "/home/inspect/app/src/inspect_ai/_eval/eval.py", line 580, in _eval_async_inner
    recorder = create_recorder_for_format(log_format or DEFAULT_LOG_FORMAT, log_dir)
  File "/home/inspect/app/src/inspect_ai/log/_recorders/create.py", line 14, in create_recorder_for_format
    return recorder(*args, **kwargs)
  File "/home/inspect/app/src/inspect_ai/log/_recorders/eval.py", line 72, in __init__
    super().__init__(log_dir, ".eval", fs_options)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/inspect/app/src/inspect_ai/log/_recorders/file.py", line 28, in __init__
    self.fs.mkdir(self.log_dir, exist_ok=True)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/inspect/app/src/inspect_ai/_util/file.py", line 194, in mkdir
    self.fs.makedir(path, create_parents=False)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/fsspec/spec.py", line 1727, in makedir
    return self.mkdir(path, create_parents=create_parents, **kwargs)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/opt/inspect_ai/lib/python3.13/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "/opt/inspect_ai/lib/python3.13/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
                ^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/s3fs/core.py", line 963, in _mkdir
    await self._ls(bucket)
  File "/opt/inspect_ai/lib/python3.13/site-packages/s3fs/core.py", line 1040, in _ls
    files = await self._lsdir(path, refresh, versions=versions)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/s3fs/core.py", line 755, in _lsdir
    async for c in self._iterdir(
    ...<9 lines>...
            files.append(c)
  File "/opt/inspect_ai/lib/python3.13/site-packages/s3fs/core.py", line 805, in _iterdir
    async for i in it:
    ...<14 lines>...
                yield c
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/paginate.py", line 30, in __anext__
    response = await self._make_request(current_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/client.py", line 394, in _make_api_call
    http, parsed_response = await self._make_request(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^
        operation_model, request_dict, request_context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/client.py", line 420, in _make_request
    return await self._endpoint.make_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        operation_model, request_dict
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/endpoint.py", line 109, in _send_request
    request = await self.create_request(request_dict, operation_model)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/endpoint.py", line 97, in create_request
    await self._event_emitter.emit(
    ...<3 lines>...
    )
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/hooks.py", line 68, in _emit
    response = await resolve_awaitable(handler(**kwargs))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/_helpers.py", line 6, in resolve_awaitable
    return await obj
           ^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/signers.py", line 24, in handler
    return await self.sign(operation_name, request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/signers.py", line 81, in sign
    auth = await self.get_auth_instance(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/signers.py", line 164, in get_auth_instance
    frozen_credentials = await credentials.get_frozen_credentials()
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/credentials.py", line 403, in get_frozen_credentials
    await self._refresh()
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/credentials.py", line 356, in _refresh
    await self._protected_refresh(
        is_mandatory=is_mandatory_refresh
    )
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/credentials.py", line 372, in _protected_refresh
    metadata = await resolve_awaitable(self._refresh_using())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/_helpers.py", line 6, in resolve_awaitable
    return await obj
           ^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/credentials.py", line 428, in fetch_credentials
    return await self._get_cached_credentials()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/credentials.py", line 438, in _get_cached_credentials
    response = await self._get_credentials()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/credentials.py", line 1049, in _get_credentials
    token = (await initial_token_data.get_frozen_token()).token
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 38, in get_frozen_token
    await self._refresh()
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 51, in _refresh
    await self._protected_refresh()
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 63, in _protected_refresh
    self._frozen_token = await self._refresh_using()
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 141, in _refresher
    new_token_dict = await self._refresh_access_token(token_dict)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 126, in _refresh_access_token
    return await self._attempt_create_token(token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/tokens.py", line 84, in _attempt_create_token
    async with self._client as client:
               ^^^^^^^^^^^^
  File "/opt/inspect_ai/lib/python3.13/site-packages/aiobotocore/session.py", line 25, in __aenter__
    self._client = await self._coro
                   ^^^^^^^^^^^^^^^^
RuntimeError: cannot reuse already awaited coroutine

What is the new behavior?

It says your credentials have expired

Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

Other information:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@revmischa