Memory leak in httpx (especially under concurrency)

[MrNaif2018]

Hi! It seems like httpx AsyncClient has memory leaks. In production, my app starts at like 60 mb, then grows very quickly up to 150 MB, and continues to grow at a slow rate (so like a week later it can be 400 MB already). My app is using httpx intensively, sending many requests in parallel via asyncio.gather (at least 20 for sure, at max it can reach a hundred or two). The RAM is not cleaned up. When isolating the use case, in an app where I was just sending a http request and nothing else, memory usage also seems to increase. This is easy to see when using asyncio.gather. That's pretty similar to #1893, but I think the case was not investigated properly. When running that bit of code for longer (or when using asyncio.gather), issue shows up instantly
At the beginning of the test, RAM usage is at ~= 30 MB. At the end of the test after like 10 minutes, it is 75 MB.
Script:

httpx testing script

import asyncio
import gc
import signal

import httpx


def getCurrentMemoryUsage() -> int:
    """Return memory usage in kB"""
    with open("/proc/self/status") as f:
        memusage = f.read().split("VmRSS:")[1].split("\n")[0][:-3]

    return int(memusage.strip())


async def memory_test(client, url):
    try:
        html = await client.get(url, follow_redirects=True)
        return html
    except Exception:
        pass


is_running = True


def signal_handler(sig, frame):
    global is_running
    is_running = False


async def main():
    start = asyncio.get_event_loop().time()
    url = "https://httpbin.org/bytes/100000"
    init_mem_usage = getCurrentMemoryUsage()
    num = 0
    async with httpx.AsyncClient() as client:
        while is_running:
            result = await asyncio.gather(*(memory_test(client, url) for _ in range(20)))
            num += 1
            print(f"Current memory usage is: {getCurrentMemoryUsage()/1000}mb")
            if asyncio.get_event_loop().time() - start >= 60 * 60 * 3:
                break

    gc.collect()
    print(f"Final memory usage: {getCurrentMemoryUsage()/1000}mb")
    print(f"Additional memory used after {num} requests: {(getCurrentMemoryUsage()-init_mem_usage)/1000}mb")


if __name__ == "__main__":
    signal.signal(signal.SIGINT, signal_handler)
    asyncio.run(main())

aiohttp testing script

import asyncio
import gc
import signal

from aiohttp import ClientSession


def getCurrentMemoryUsage() -> int:
    """Return memory usage in kB"""
    with open("/proc/self/status") as f:
        memusage = f.read().split("VmRSS:")[1].split("\n")[0][:-3]

    return int(memusage.strip())


# @profile(precision=4)
async def memory_test(client, url):
    try:
        async with client.get(url) as resp:
            return await resp.read()
    except Exception:
        pass


is_running = True


def signal_handler(sig, frame):
    global is_running
    is_running = False


async def main():
    start = asyncio.get_event_loop().time()
    url = "https://httpbin.org/bytes/100000"
    init_mem_usage = getCurrentMemoryUsage()
    num = 0
    async with ClientSession() as client:
        while is_running:
            result = await asyncio.gather(*(memory_test(client, url) for _ in range(20)))
            num += 1
            print(f"Current memory usage is: {getCurrentMemoryUsage()/1000}mb")
            if asyncio.get_event_loop().time() - start >= 60 * 60 * 3:
                break

    gc.collect()
    print(f"Final memory usage: {getCurrentMemoryUsage()/1000}mb")
    print(f"Additional memory used after {num} requests: {(getCurrentMemoryUsage()-init_mem_usage)/1000}mb")


if __name__ == "__main__":
    signal.signal(signal.SIGINT, signal_handler)
    asyncio.run(main())

Screenshots somewhere in the middle of the test:
For httpx:

For aiohttp (see that it is steady, though it increases a little bit too which is weird):

[powellnorma]

In my case, I am streaming data files via async with session.stream, and after some days/weeks I see the app taking ~8GB of ram.

I tried to find the leak via https://pypi.org/project/mem-top/, but it showed nothing suspicious. Maybe someone knows another library one could use to check for memory leaks?

[peterlandry]

SSL context setup seems to be able to leak memory.