[Bug]: Memory leak from Python

[glowww]

TLS client version

1.8.0

System information

MacBook Pro M3 Pro
macOS 15.3.1

Issue description

When making requests that return large responses, memory usage goes through the roof. It doesn’t seem to scale properly with the response size.

Output

Before request: 15MB
After request: 700MB
Response size: 82MB
After response: 1342MB
After freeMemory: 1342MB

A 82MB response shouldn’t cause memory usage to jump by over 1GB. Some increase is expected, but this seems excessive.

Steps to reproduce / Code Sample

import ctypes
import json

import psutil


def memory_usage():
    return psutil.Process().memory_info().rss // (1024 ** 2)


library = ctypes.cdll.LoadLibrary('../lib/tls/tls-client-darwin-arm64-1.8.0.dylib')

# extract the exposed request function from the shared package
request = library.request
request.argtypes = [ctypes.c_char_p]
request.restype = ctypes.c_char_p

freeMemory = library.freeMemory
freeMemory.argtypes = [ctypes.c_char_p]

requestPayload = {
    "tlsClientIdentifier": "chrome_105",
    "requestUrl": "https://platform.tonybet.com/api/event/list?limit=200&main=0&page=1&oddsExists_eq=1&period=0&lang=en&relations%5B%5D=odds&relations%5B%5D=competitors&relations%5B%5D=players&relations%5B%5D=variants&relations%5B%5D=league&relations%5B%5D=sport",
    "requestMethod": "GET",
}

print(f"Before request: {memory_usage()}MB")

response = request(json.dumps(requestPayload).encode('utf-8'))

print(f"After request: {memory_usage()}MB")

response_bytes = ctypes.string_at(response)
response_string = response_bytes.decode('utf-8')
response_object = json.loads(response_string)

print(f"Response size: {len(response_bytes) // (1024 ** 2)}MB")

print(f"After response: {memory_usage()}MB")

freeMemory(response_object['id'].encode('utf-8'))

print(f"After freeMemory: {memory_usage()}MB")

[bogdanfinn]

@glowww thank you very much for writing this issue and providing code to reproduce the issue.

To be honest i totally have no clue about garbage collection in python. But as far as i know a lot of people are using this client in python and nobody really complains about memory leaks.

First of all the tls-client itself consumes a bit of memory - thats for sure. But there should not be a memory leak.
My "assumption" is:

a) that your code is done that fast, that the garbage collection of python does not kick in time
b) the garbage collection is not freeing any memory because you handle everything in the "main process" and there is still reference on the variables and stuff.

If we rewrite your code a bit to run the memory heavy logic in a separate function all references to the variables used in the function should be gone and the gargabe collection can cleanup the memory.

Note that i had to change the library loading path

import ctypes
import json
import psutil

library = ctypes.cdll.LoadLibrary('../dist/tls-client-xgo-1.8.0-linux-amd64.so')

# extract the exposed request function from the shared package
request = library.request
request.argtypes = [ctypes.c_char_p]
request.restype = ctypes.c_char_p

freeMemory = library.freeMemory
freeMemory.argtypes = [ctypes.c_char_p]

def memory_usage():
    return psutil.Process().memory_info().rss // (1024 ** 2)

def do_request():
    print(f"Before request: {memory_usage()}MB")

    requestPayload = {
        "tlsClientIdentifier": "chrome_105",
        "requestUrl": "https://platform.tonybet.com/api/event/list?limit=200&main=0&page=1&oddsExists_eq=1&period=0&lang=en&relations%5B%5D=odds&relations%5B%5D=competitors&relations%5B%5D=players&relations%5B%5D=variants&relations%5B%5D=league&relations%5B%5D=sport",
        "requestMethod": "GET",
    }

    response = request(json.dumps(requestPayload).encode('utf-8'))

    print(f"After request: {memory_usage()}MB")

    response_bytes = ctypes.string_at(response)
    print(f"Response size: {len(response_bytes) // (1024 ** 2)}MB")

    response_string = response_bytes.decode('utf-8')
    response_object = json.loads(response_string)

    print(f"After response: {memory_usage()}MB")

    freeMemory(response_object['id'].encode('utf-8'))

    print(f"After freeMemory: {memory_usage()}MB")


print(f"Before function call: {memory_usage()}MB")
do_request()
print(f"After function call: {memory_usage()}MB")

Output

Before function call: 20MB
Before request: 20MB
After request: 726MB
Response size: 83MB
After response: 1124MB
After freeMemory: 1041MB
After function call: 559MB

We see that most of the memory is being released. Of course there is still allocated memory but the point is now, that it should not leak if we do multiple requests:

Note: we are doing 5 requests after each other and on purpose are not using tls-client sessions

import ctypes
import json
import psutil

library = ctypes.cdll.LoadLibrary('../dist/tls-client-xgo-1.8.0-linux-amd64.so')

# extract the exposed request function from the shared package
request = library.request
request.argtypes = [ctypes.c_char_p]
request.restype = ctypes.c_char_p

freeMemory = library.freeMemory
freeMemory.argtypes = [ctypes.c_char_p]

def memory_usage():
    return psutil.Process().memory_info().rss // (1024 ** 2)

def do_request():
    print(f"Before request: {memory_usage()}MB")

    requestPayload = {
        "tlsClientIdentifier": "chrome_105",
        "requestUrl": "https://platform.tonybet.com/api/event/list?limit=200&main=0&page=1&oddsExists_eq=1&period=0&lang=en&relations%5B%5D=odds&relations%5B%5D=competitors&relations%5B%5D=players&relations%5B%5D=variants&relations%5B%5D=league&relations%5B%5D=sport",
        "requestMethod": "GET",
    }

    response = request(json.dumps(requestPayload).encode('utf-8'))

    print(f"After request: {memory_usage()}MB")

    response_bytes = ctypes.string_at(response)
    print(f"Response size: {len(response_bytes) // (1024 ** 2)}MB")

    response_string = response_bytes.decode('utf-8')
    response_object = json.loads(response_string)

    print(f"After response: {memory_usage()}MB")

    freeMemory(response_object['id'].encode('utf-8'))

    print(f"After freeMemory: {memory_usage()}MB")


print(f"Before 1. function call: {memory_usage()}MB")
do_request()
print(f"Before 2. function call: {memory_usage()}MB")
do_request()
print(f"Before 3. function call: {memory_usage()}MB")
do_request()
print(f"Before 4. function call: {memory_usage()}MB")
do_request()
print(f"Before 5. function call: {memory_usage()}MB")
do_request()
print(f"After last function call: {memory_usage()}MB")

Output:

Before 1. function call: 20MB
Before request: 20MB
After request: 722MB
Response size: 83MB
After response: 1121MB
After freeMemory: 1037MB
Before 2. function call: 555MB
Before request: 555MB
After request: 729MB
Response size: 83MB
After response: 1127MB
After freeMemory: 1043MB
Before 3. function call: 562MB
Before request: 562MB
After request: 742MB
Response size: 83MB
After response: 1141MB
After freeMemory: 1057MB
Before 4. function call: 575MB
Before request: 575MB
After request: 728MB
Response size: 83MB
After response: 1127MB
After freeMemory: 1043MB
Before 5. function call: 561MB
Before request: 561MB
After request: 730MB
Response size: 83MB
After response: 1128MB
After freeMemory: 1044MB
After last function call: 563MB

We see that the memory stays more or less stable. Of course this depends on how long you are running the code i guess because at leat the garbage collection of go needs some time to kick in.

Again i have absolute no clue about garbage collection in python.

But from what i can say there is no real memory leak but just a bit more allocated memory which might be fine depending on your scale.

[glowww]

Thanks for your detailed response @bogdanfinn and for taking the time to test this! You’re absolutely right—the memory does seem to stabilize after some time, and garbage collection eventually kicks in. However, this is still a major issue when making multiple requests concurrently.

When running multiple requests in parallel, the memory usage skyrockets to absurd levels because Python is too slow to release memory. This becomes a serious bottleneck in high-load scenarios. Unfortunately, I’m not sure what the best approach is to handle this. Do you think there’s any room for optimization to avoid this? The memory usage for a single request is very high and doesn’t seem to add up to even 10% of the response size.

[samstone1998]

So just my small input on this - I have also seen what I believe are memory leaks in the past but I ended up using the API client anyway so it didn't impact me in the long term.

So using context managers and separate functions can indeed reduce memory leaks in general. The issue is ctypes are not stored in the python heap and are not native python objects. So even when the garbage collector has met its thresholds and decided to run it will never free up ctype memory as really it don't even know about it.

If you want to test garbage collection you can do the following

import gc

def test_callback(phase, info):
    print("Garbage collection phase:", phase)
    print("Garbage collection info:", info)

gc.callbacks.append(test_callback)

# Put this in the location you want little garbage man to run
gc.collect()

Example output (showing 17 objects were collected)

Garbage collection phase: start
Garbage collection info: {'generation': 2, 'collected': 0, 'uncollectable': 0}
Garbage collection phase: stop
Garbage collection info: {'generation': 2, 'collected': 17, 'uncollectable': 0}

I am able to replicate the huge memory increase on M4 mac but am yet to be able to work out how to free that memory.