-
-
Notifications
You must be signed in to change notification settings - Fork 999
Closed
Labels
bugSomething isn't workingSomething isn't workingpoolingIssues and PRs relative to connection poolingIssues and PRs relative to connection pooling
Description
Hello. I am having an issue where it looks like connections aren't being closed correctly, and after i reach a number of requests equivalent to "hard_limit" of pool_limits, i get a PoolTimeout exception.
I tried upgrading to httpx==0.10.1, with no success.
Minimal example:
import httpx, asyncio, logging
from httpx import PoolLimits
from random import randint
queue = asyncio.Queue()
clients = [
httpx.AsyncClient(
http2=True,
pool_limits=PoolLimits(soft_limit=2, hard_limit=10),
cookies={'a': '123456789', 'b': '987654321'},
)
]
async def worker_loop(cid, client, queue):
while 1:
sub_id = await queue.get()
async with client as c:
r = await c.get(f'https://mywebsite.dummy/submission.php?id={sub_id}')
if r.status_code != 200:
print(cid, f'Got status code {r.status_code} while parsing {sub_id}')
return
async def main():
for i in range(2500):
await queue.put(randint(1, 80000000))
for k, v in enumerate(clients):
asyncio.create_task(worker_loop(k, v, queue))
while 1:
if queue.qsize() == 0:
await queue.put(randint(1, 80000000))
await asyncio.sleep(2)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.stop()
I checked with netstat, and only one actual connection is opened to the IP address, so pooling seems to work fine.
I really cannot understand why. I even tried using the "aclose()" syntax, without the "async with" block, but no difference at all.
devxpy
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingpoolingIssues and PRs relative to connection poolingIssues and PRs relative to connection pooling