Unable to release memory

[msoltysik]

Describe your environment

OS: Darwin Kernel Version 23.6.0
Python version: 3.12.6
SDK version: 1.27.0
API version: 1.27.0

What happened?

I've confirmed the issue with Aaron Abbott. More details can be found in the slack thread.

In a nutshell, I am unable to release the memory of otlp_exporter and reader.

Steps to Reproduce

I've prepared a code snippet that demonstrates the problem:

1. Initialize otlp_exporter, reader, provider with weakref trackers
2. Cleanup using .shutdown(), del and gc.collect()
3. Assert if the weakref referneces are None

Expected Result

2024-10-10 07:37:18,766 INFO [root] [reproduce.py:38] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - OTLP EXPORTER was properly garbage collected
2024-10-10 07:37:18,766 INFO [root] [reproduce.py:47] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - READER was properly garbage collected
2024-10-10 07:37:18,767 INFO [root] [reproduce.py:62] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - PROVIDER was properly garbage collected

Actual Result

2024-10-10 07:37:18,761 WARNING [opentelemetry.sdk.metrics._internal.export] [__init__.py:556] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - Can't shutdown multiple times
2024-10-10 07:37:18,766 WARNING [root] [reproduce.py:38] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - OTLP EXPORTER was not properly garbage collected
2024-10-10 07:37:18,766 DEBUG [root] [reproduce.py:40] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - Direct referrers to OTLP EXPORTER: 4
2024-10-10 07:37:18,766 DEBUG [root] [reproduce.py:42] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - OTLP EXPORTER referrer {<opentelemetry.sdk.metrics._internal.export.PeriodicExportingMetricReader object at 0x1063d4d10>} type: <class 'set'>
2024-10-10 07:37:18,766 DEBUG [root] [reproduce.py:42] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - OTLP EXPORTER referrer <bound method PeriodicExportingMetricReader._at_fork_reinit of <opentelemetry.sdk.metrics._internal.export.PeriodicExportingMetricReader object at 0x1063d4d10>> type: <class 'method'>
2024-10-10 07:37:18,766 DEBUG [root] [reproduce.py:42] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - OTLP EXPORTER referrer {<opentelemetry.sdk.metrics._internal.export.PeriodicExportingMetricReader object at 0x1063d4d10>: <opentelemetry.sdk.metrics._internal.metric_reader_storage.MetricReaderStorage object at 0x1063d5520>} type: <class 'dict'>
2024-10-10 07:37:18,766 DEBUG [root] [reproduce.py:42] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - OTLP EXPORTER referrer [<opentelemetry.sdk.metrics._internal.export.PeriodicExportingMetricReader object at 0x1063d4d10>] type: <class 'list'>
2024-10-10 07:37:18,766 WARNING [root] [reproduce.py:47] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - READER was not properly garbage collected
2024-10-10 07:37:18,767 DEBUG [root] [reproduce.py:49] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - Direct referrers to READER: 4
2024-10-10 07:37:18,767 DEBUG [root] [reproduce.py:51] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - READER referrer {<opentelemetry.sdk.metrics._internal.export.PeriodicExportingMetricReader object at 0x1063d4d10>} type: <class 'set'>
2024-10-10 07:37:18,767 DEBUG [root] [reproduce.py:51] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - READER referrer <bound method PeriodicExportingMetricReader._at_fork_reinit of <opentelemetry.sdk.metrics._internal.export.PeriodicExportingMetricReader object at 0x1063d4d10>> type: <class 'method'>
2024-10-10 07:37:18,767 DEBUG [root] [reproduce.py:51] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - READER referrer {<opentelemetry.sdk.metrics._internal.export.PeriodicExportingMetricReader object at 0x1063d4d10>: <opentelemetry.sdk.metrics._internal.metric_reader_storage.MetricReaderStorage object at 0x1063d5520>} type: <class 'dict'>
2024-10-10 07:37:18,767 DEBUG [root] [reproduce.py:51] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - READER referrer [<opentelemetry.sdk.metrics._internal.export.PeriodicExportingMetricReader object at 0x1063d4d10>] type: <class 'list'>
2024-10-10 07:37:18,767 INFO [root] [reproduce.py:62] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - PROVIDER was properly garbage collected

Additional context

No response

Would you like to implement a fix?

No

[aabmass]

2024-10-10 07:37:18,766 DEBUG [root] [reproduce.py:42] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - OTLP EXPORTER referrer <bound method PeriodicExportingMetricReader._at_fork_reinit of <opentelemetry.sdk.metrics._internal.export.PeriodicExportingMetricReader object at 0x1063d4d10>> type: <class 'method'>

I'm hoping this is the problematic "root" reference and the other referrers are part of a subgraph. I'm thinking it is caused by this post fork hook holding a reference to the class

opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/trace/export/__init__.py

Lines 206 to 209 in 2bcbbcc

	if hasattr(os, "register_at_fork"):
	os.register_at_fork(
	after_in_child=self._at_fork_reinit
	) # pylint: disable=protected-access

Unfortunately there is no way to unregister a hook at shutdown. I wonder if commenting out this code would fix the problem. If that works, maybe we can call through a weakref.

[aabmass]

@msoltysik thanks for the report, I figured it out.

In addition to the post fork hook I mentioned above, there was also a leaked reference in this class variable set

opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/__init__.py

Line 396 in 2bcbbcc

_all_metric_readers = set()

That check was added IIRC to satisfy the metric specification not allowing a metric reader to be registered to more than one MeterProvider.

Here is a PoC fix branch main...aabmass:fix-gc-issues. I don't have time to write tests, would you be willing to take over that branch @msoltysik?

[msoltysik]

Thanks @aabmass, I'll take over and prepare a PR with tests early next week.