Potential memory leak: CIMultiDict objects never garbage collected

[hoenie-ams]

Please confirm the following

• I agree to follow the aio-libs Code of Conduct

• I have checked the current issues for duplicates.

• I understand this is open source software provided for free and that I might not receive a timely response.
• I am positive I am NOT reporting a (potential) security
  vulnerability, to the best of my knowledge. (These must be shared by
  submitting this report form instead, if
  any hesitation exists.)

• I am willing to submit a pull request with reporoducers as xfailing test cases or even entire fix. (Assign this issue to me.)

Describe the bug

First of all, thanks for all your amazing work on open source!

Hopefully, I am at the right place for this issue, but it might belong in another package (like multidict, aiohttp, scrapy or scrapy-zyte-api). The issue I encountered happened exactly when 0.4.0 got released though.

So after upgrading from propcache 0.3.2 to 0.4.0, CIMultiDict objects
from aiohttp HTTP headers accumulate in memory and are never released.
gc.collect() has no effect, indicating strong references are being held.

I suspect that changes in src/propcache/_helpers_c.pyx may be caching
properties that reference these objects without proper cleanup.

To Reproduce

Sorry I don't have a copy/past example, it is part of a scrapy project that I cannot share unfortunately. But here are some details:

• Run long-running aiohttp client (via scrapy-zyte-api in my case)
• Process thousands of HTTP requests
• Monitor: sum(1 for obj in gc.get_objects() if type(obj).__name__ == 'CIMultiDict')
• Count grows linearly with requests and never decreases

Packages used:

• propcache: 0.4.0
• aiohttp: 3.13.0
• multidict: 6.7.0

Expected behavior

Expected: CIMultiDict objects should be garbage collected after use
Actual: They accumulate indefinitely (~1 per request)

Logs/tracebacks

N.A.

Python Version

3.13

propcache Version

0.4.0

OS

Linux

Additional context

Workaround: Pin to propcache==0.3.2, multidict==6.6.3 & aiohttp==3.12.14 solved it

[bdraco]

Thanks for reporting this! To help us investigate and fix this issue, could you provide some additional information?

Which specific pin resolves the issue?

You mentioned pinning to older versions as a workaround. Could you test pinning each package individually to isolate which one is causing the problem?

• Does pinning only propcache==0.3.0 (while keeping multidict 6.7.0) resolve it?
• Does pinning only multidict==6.6.0 (while keeping propcache 0.4.0) resolve it?

This will help us determine whether the issue is in propcache or multidict.

Could you provide a minimal script that reproduces the memory leak? This should be a complete, runnable script that demonstrates the issue without requiring external dependencies like scrapy or a full application setup.

Without a way to reproduce the issue locally, it will be very difficult for us to identify and fix the root cause.

Thanks for your help! 🙏

[skgsergio]

We started experiencing a memory leak on Monday. After checking the dependencies we had updated and analyzing the issue using memray, we've found this issue in GitHub.

In our case, we did not update multidict, it’s pinned to 6.6.4, but we did update propcache from 0.3.2 to 0.4.0.

[bdraco]

cc @Vizonex (#121)

Looks like a ref leak

[Tasssadar]

Looks like the <object>val might be increasing the ref count by itself, so the extra Py_INCREF(val) that is in code could be creating extra reference.

Fix could be this, you probably have the development environment ready and it will be quicker for you to do it:

diff --git a/src/propcache/_helpers_c.pyx b/src/propcache/_helpers_c.pyx
index 72167d3..35cce03 100644
--- a/src/propcache/_helpers_c.pyx
+++ b/src/propcache/_helpers_c.pyx
@@ -17,6 +17,7 @@ cdef extern from "Python.h":
         object dict, object key, PyObject* value
     ) except -1
     void Py_INCREF(PyObject*)
+    void Py_DECREF(PyObject*)
 
 cdef class under_cached_property:
     """Use as a class method decorator.  It operates almost exactly like
@@ -43,12 +44,16 @@ cdef class under_cached_property:
             return self
         cdef dict cache = inst._cache
         cdef PyObject* val = PyDict_GetItem(cache, self.name)
-        if val == NULL:
-            val = PyObject_CallOneArg(self.wrapped, inst)
-            PyDict_SetItem(cache, self.name, val)
-        else:
-            Py_INCREF(val)
-        return <object>val
+        if val is not NULL:
+            return <object>val
+
+        val = PyObject_CallOneArg(self.wrapped, inst)
+        PyDict_SetItem(cache, self.name, val)
+        # The <object> cast increments ref count,
+        # so we need to decref the original
+        val_obj = <object>val
+        Py_DECREF(val)
+        return val_obj
 
     def __set__(self, inst, value):
         raise AttributeError("cached property is read-only")
@@ -94,11 +99,15 @@ cdef class cached_property:
                 " without calling __set_name__ on it.")
         cdef dict cache = inst.__dict__
         cdef PyObject* val = PyDict_GetItem(cache, self.name)
-        if val is NULL:
-            val = PyObject_CallOneArg(self.func, inst)
-            PyDict_SetItem(cache, self.name, val)
-        else:
-            Py_INCREF(val)
-        return <object>val
+        if val is not NULL:
+            return <object>val
+
+        val = PyObject_CallOneArg(self.func, inst)
+        PyDict_SetItem(cache, self.name, val)
+        # The <object> cast increments ref count,
+        # so we need to decref the original
+        val_obj = <object>val
+        Py_DECREF(val)
+        return val_obj
 

[hoenie-ams]

@bdraco, Sorry for not providing the MRE already. It was getting quite late last night. Here is one:

import asyncio
import gc

import aiohttp


async def main():
    async with aiohttp.ClientSession() as session:
        for _ in range(5):
            async with session.get('https://httpbin.org/get') as response:
                count = sum(1 for obj in gc.get_objects() if type(obj).__name__ == 'CIMultiDict')
                print(f"Count of CIMultiDict objects: {count}")


if __name__ == '__main__':
    asyncio.run(main())

Run with propcache==0.3.2

Count of CIMultiDict objects: 3
Count of CIMultiDict objects: 3
Count of CIMultiDict objects: 3
Count of CIMultiDict objects: 3
Count of CIMultiDict objects: 3

Run with propcache==0.4.0

Count of CIMultiDict objects: 3
Count of CIMultiDict objects: 4
Count of CIMultiDict objects: 5
Count of CIMultiDict objects: 6
Count of CIMultiDict objects: 7

[Vizonex]

cc @Vizonex (#121)

Looks like a ref leak

Crap :( Whelp Looks like I'm gonna have to fix it. Edit I thought you were the one that wanted me to up the ref counter by 1

[Vizonex]

@hoenie-ams I apologize for the 19 hour wait I should've been more responsive about what is internally happening but basically when I made my original PR the maintainers had the worry that I could be a triggering crash, that was when I was asked to add in Py_INCREF Don't worry I am coming up with a fix so that hopefully this doesn't happen again.

Edit: I don't like this either and I'm on the same boat as you, Reminds me of the recent memory leak that was happening with multidict where clearing values would trigger memory leaks if the hashtable was 0.

[Vizonex]

@hoenie-ams Feel free to run my newer fork https://github.com/Vizonex/propcache/tree/fix-ref-leak and see if it stops the problem.

[bdraco]

I yanked 0.4.0 in the mean time