[Debugger Default-On] DEBUG-3322 Debugger in-product enablement #7366

dudikeleti · 2025-08-07T23:13:55Z

Summary of changes

This PR adds remote enablement support for debugger products.
depends on #7441

Reason for change

Currently, the only option to enable DI (for example) is by setting an environment variable. We want to give customers the ability to enable products on demand.

Implementation details

Gets RC events with new configuration and update the products based on defined rules.

Test coverage

All existing tests
New system tests (DataDog/system-tests#4991)
DebuggerManagerTests
DebuggerManagerDynamicTests

dd-trace-dotnet-ci-bot · 2025-08-07T23:55:57Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7366) - mean (72ms)  : 71, 73
     .   : milestone, 72,
    master - mean (72ms)  : 71, 73
     .   : milestone, 72,

    section Baseline
    This PR (7366) - mean (68ms)  : 67, 70
     .   : milestone, 68,
    master - mean (68ms)  : 66, 70
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (7366) - mean (1,052ms)  : 992, 1113
     .   : milestone, 1052,
    master - mean (1,061ms)  : 975, 1147
     .   : milestone, 1061,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7366) - mean (106ms)  : 105, 107
     .   : milestone, 106,
    master - mean (107ms)  : 105, 108
     .   : milestone, 107,

    section Baseline
    This PR (7366) - mean (105ms)  : 103, 108
     .   : milestone, 105,
    master - mean (106ms)  : 104, 108
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (7366) - mean (744ms)  : 724, 764
     .   : milestone, 744,
    master - mean (744ms)  : 717, 772
     .   : milestone, 744,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7366) - mean (101ms)  : 100, 102
     .   : milestone, 101,
    master - mean (100ms)  : 99, 102
     .   : milestone, 100,

    section Baseline
    This PR (7366) - mean (100ms)  : 98, 102
     .   : milestone, 100,
    master - mean (100ms)  : 97, 103
     .   : milestone, 100,

    section CallTarget+Inlining+NGEN
    This PR (7366) - mean (782ms)  : 742, 821
     .   : milestone, 782,
    master - mean (777ms)  : 741, 814
     .   : milestone, 777,

gantt
    title Execution time (ms) FakeDbCommand (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7366) - mean (93ms)  : 92, 94
     .   : milestone, 93,
    master - mean (93ms)  : 91, 94
     .   : milestone, 93,

    section Baseline
    This PR (7366) - mean (92ms)  : 90, 95
     .   : milestone, 92,
    master - mean (92ms)  : 90, 94
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (7366) - mean (659ms)  : 643, 674
     .   : milestone, 659,
    master - mean (665ms)  : 648, 683
     .   : milestone, 665,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7366) - mean (198ms)  : 195, 201
     .   : milestone, 198,
    master - mean (201ms)  : 197, 204
     .   : milestone, 201,

    section Baseline
    This PR (7366) - mean (195ms)  : 190, 200
     .   : milestone, 195,
    master - mean (196ms)  : 190, 202
     .   : milestone, 196,

    section CallTarget+Inlining+NGEN
    This PR (7366) - mean (1,173ms)  : 1116, 1230
     .   : milestone, 1173,
    master - mean (1,177ms)  : 1110, 1243
     .   : milestone, 1177,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7366) - mean (282ms)  : 276, 288
     .   : milestone, 282,
    master - mean (282ms)  : 275, 289
     .   : milestone, 282,

    section Baseline
    This PR (7366) - mean (281ms)  : 276, 286
     .   : milestone, 281,
    master - mean (282ms)  : 275, 289
     .   : milestone, 282,

    section CallTarget+Inlining+NGEN
    This PR (7366) - mean (937ms)  : 894, 980
     .   : milestone, 937,
    master - mean (942ms)  : 895, 989
     .   : milestone, 942,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7366) - mean (285ms)  : 279, 291
     .   : milestone, 285,
    master - mean (284ms)  : 278, 289
     .   : milestone, 284,

    section Baseline
    This PR (7366) - mean (284ms)  : 279, 288
     .   : milestone, 284,
    master - mean (284ms)  : 279, 288
     .   : milestone, 284,

    section CallTarget+Inlining+NGEN
    This PR (7366) - mean (1,007ms)  : 967, 1046
     .   : milestone, 1007,
    master - mean (1,003ms)  : 960, 1046
     .   : milestone, 1003,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7366) - mean (271ms)  : 266, 277
     .   : milestone, 271,
    master - mean (273ms)  : 269, 277
     .   : milestone, 273,

    section Baseline
    This PR (7366) - mean (271ms)  : 267, 275
     .   : milestone, 271,
    master - mean (271ms)  : 266, 276
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (7366) - mean (854ms)  : 837, 871
     .   : milestone, 854,
    master - mean (862ms)  : 833, 890
     .   : milestone, 862,

pr-commenter · 2025-08-08T00:16:02Z

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmarks for #7366 compared to master:

2 benchmarks are faster, with geometric mean 1.893
3 benchmarks have fewer allocations
3 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	10.6μs	60ns	384ns	0	0	0	5.49 KB
master	`StartStopWithChild`	netcoreapp3.1	13.5μs	69.9ns	342ns	0	0	0	5.71 KB
master	`StartStopWithChild`	net472	22.7μs	121ns	641ns	0.955	0.318	0.106	6 KB
#7366	`StartStopWithChild`	net6.0	11.4μs	55.3ns	228ns	0	0	0	5.48 KB
#7366	`StartStopWithChild`	netcoreapp3.1	13.4μs	65ns	283ns	0	0	0	5.69 KB
#7366	`StartStopWithChild`	net472	21.8μs	106ns	448ns	0.98	0.436	0.109	5.98 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	932μs	76.3ns	264ns	2.71 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	1.03ms	96.1ns	360ns	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	1.2ms	2.36μs	9.15μs	3.31 KB
#7366	`WriteAndFlushEnrichedTraces`	net6.0	936μs	87.7ns	340ns	2.72 KB
#7366	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	1.01ms	36.7ns	132ns	2.71 KB
#7366	`WriteAndFlushEnrichedTraces`	net472	1.19ms	48.8ns	176ns	3.31 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`AllCycleSimpleBody`	net6.0	328μs	312ns	1.21μs	0	172.08 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	457μs	1.16μs	4.51μs	0	174.18 KB
master	`AllCycleSimpleBody`	net472	428μs	144ns	558ns	29.2	194.24 KB
master	`AllCycleMoreComplexBody`	net6.0	335μs	912ns	3.41μs	0	175.58 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	503μs	872ns	3.38μs	0	177.6 KB
master	`AllCycleMoreComplexBody`	net472	440μs	145ns	560ns	30.2	197.76 KB
master	`ObjectExtractorSimpleBody`	net6.0	329ns	1.58ns	6.7ns	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	398ns	2.08ns	10.8ns	0	272 B
master	`ObjectExtractorSimpleBody`	net472	300ns	0.188ns	0.726ns	0.0437	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	6.29μs	30ns	116ns	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	7.76μs	35.1ns	136ns	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	6.74μs	5.78ns	22.4ns	0.57	3.8 KB
#7366	`AllCycleSimpleBody`	net6.0	326μs	292ns	1.13μs	0	172.07 KB
#7366	`AllCycleSimpleBody`	netcoreapp3.1	469μs	412ns	1.6μs	0	174.18 KB
#7366	`AllCycleSimpleBody`	net472	431μs	130ns	504ns	30.2	194.25 KB
#7366	`AllCycleMoreComplexBody`	net6.0	333μs	1.51μs	5.85μs	0	175.58 KB
#7366	`AllCycleMoreComplexBody`	netcoreapp3.1	485μs	1.98μs	7.67μs	0	177.6 KB
#7366	`AllCycleMoreComplexBody`	net472	437μs	212ns	821ns	30.2	197.76 KB
#7366	`ObjectExtractorSimpleBody`	net6.0	317ns	0.0998ns	0.374ns	0	280 B
#7366	`ObjectExtractorSimpleBody`	netcoreapp3.1	396ns	2.22ns	13.9ns	0	272 B
#7366	`ObjectExtractorSimpleBody`	net472	299ns	0.0312ns	0.117ns	0.044	281 B
#7366	`ObjectExtractorMoreComplexBody`	net6.0	6.3μs	28.7ns	115ns	0	3.78 KB
#7366	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	7.71μs	32.6ns	126ns	0	3.69 KB
#7366	`ObjectExtractorMoreComplexBody`	net472	6.74μs	9.7ns	37.6ns	0.577	3.8 KB

Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EncodeArgs`	net6.0	77.1μs	53.4ns	207ns	0	32.4 KB
master	`EncodeArgs`	netcoreapp3.1	97μs	245ns	948ns	0	32.4 KB
master	`EncodeArgs`	net472	109μs	48.1ns	180ns	4.93	32.5 KB
master	`EncodeLegacyArgs`	net6.0	145μs	103ns	401ns	0	2.15 KB
master	`EncodeLegacyArgs`	netcoreapp3.1	197μs	203ns	787ns	0	2.14 KB
master	`EncodeLegacyArgs`	net472	262μs	66.4ns	257ns	0	2.16 KB
#7366	`EncodeArgs`	net6.0	77.8μs	24.9ns	89.8ns	0	32.4 KB
#7366	`EncodeArgs`	netcoreapp3.1	97.8μs	218ns	815ns	0	32.4 KB
#7366	`EncodeArgs`	net472	109μs	18.7ns	64.7ns	4.93	32.5 KB
#7366	`EncodeLegacyArgs`	net6.0	145μs	290ns	1.12μs	0	2.15 KB
#7366	`EncodeLegacyArgs`	netcoreapp3.1	196μs	186ns	720ns	0	2.14 KB
#7366	`EncodeLegacyArgs`	net472	263μs	96.8ns	362ns	0	2.16 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #7366

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark‑netcoreapp3.1	2.077	868,931.67	418,413.90
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack‑netcoreapp3.1	1.726	518,038.75	300,215.18

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`RunWafRealisticBenchmark`	net6.0	398μs	161ns	622ns	4.55 KB
master	`RunWafRealisticBenchmark`	netcoreapp3.1	817μs	14.7μs	144μs	4.48 KB
master	`RunWafRealisticBenchmark`	net472	437μs	80.5ns	301ns	4.66 KB
master	`RunWafRealisticBenchmarkWithAttack`	net6.0	289μs	106ns	367ns	2.24 KB
master	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	490μs	6.83μs	67.9μs	2.22 KB
master	`RunWafRealisticBenchmarkWithAttack`	net472	313μs	58.8ns	212ns	2.29 KB
#7366	`RunWafRealisticBenchmark`	net6.0	401μs	423ns	1.64μs	4.55 KB
#7366	`RunWafRealisticBenchmark`	netcoreapp3.1	419μs	374ns	1.45μs	4.48 KB
#7366	`RunWafRealisticBenchmark`	net472	436μs	92.9ns	360ns	4.66 KB
#7366	`RunWafRealisticBenchmarkWithAttack`	net6.0	290μs	180ns	697ns	2.24 KB
#7366	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	300μs	243ns	940ns	2.22 KB
#7366	`RunWafRealisticBenchmarkWithAttack`	net472	313μs	147ns	568ns	2.29 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`SendRequest`	net6.0	61μs	51.2ns	177ns	14.52 KB
master	`SendRequest`	netcoreapp3.1	71.2μs	67.2ns	260ns	17.42 KB
master	`SendRequest`	net472	0.00966ns	0.00336ns	0.013ns	0 b
#7366	`SendRequest`	net6.0	60.1μs	73.8ns	266ns	14.52 KB
#7366	`SendRequest`	netcoreapp3.1	70.1μs	79.3ns	307ns	17.42 KB
#7366	`SendRequest`	net472	0.00977ns	0.00305ns	0.0118ns	0 b

Benchmarks.Trace.CharSliceBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7366

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net6.0	4 B	7 B	3 B	75.00%

Fewer allocations 🎉 in #7366

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net472	49 B	47 B	-2 B	-4.08%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net6.0	5 B	3 B	-2 B	-40.00%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`OriginalCharSlice`	net6.0	1.9ms	4.67μs	18.1μs	0	640.01 KB
master	`OriginalCharSlice`	netcoreapp3.1	2.08ms	7.43μs	26.8μs	0	640 KB
master	`OriginalCharSlice`	net472	2.64ms	578ns	2.24μs	100	641.95 KB
master	`OptimizedCharSlice`	net6.0	1.49ms	99.8ns	386ns	0	4 B
master	`OptimizedCharSlice`	netcoreapp3.1	1.64ms	579ns	2.24μs	0	1 B
master	`OptimizedCharSlice`	net472	1.96ms	324ns	1.21μs	0	73 B
master	`OptimizedCharSliceWithPool`	net6.0	796μs	30ns	116ns	0	5 B
master	`OptimizedCharSliceWithPool`	netcoreapp3.1	832μs	72.6ns	262ns	0	1 B
master	`OptimizedCharSliceWithPool`	net472	1.18ms	241ns	902ns	0	49 B
#7366	`OriginalCharSlice`	net6.0	1.95ms	599ns	2.24μs	0	640.01 KB
#7366	`OriginalCharSlice`	netcoreapp3.1	2.13ms	2.78μs	10.8μs	0	640 KB
#7366	`OriginalCharSlice`	net472	2.6ms	274ns	1.06μs	100	641.95 KB
#7366	`OptimizedCharSlice`	net6.0	1.35ms	568ns	2.2μs	0	7 B
#7366	`OptimizedCharSlice`	netcoreapp3.1	1.68ms	323ns	1.21μs	0	1 B
#7366	`OptimizedCharSlice`	net472	1.97ms	520ns	2.01μs	0	73 B
#7366	`OptimizedCharSliceWithPool`	net6.0	835μs	32.1ns	120ns	0	3 B
#7366	`OptimizedCharSliceWithPool`	netcoreapp3.1	844μs	146ns	564ns	0	1 B
#7366	`OptimizedCharSliceWithPool`	net472	1.16ms	90.6ns	339ns	0	47 B

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	683μs	1.75μs	6.07μs	0	41.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	680μs	745ns	2.88μs	0	41.87 KB
master	`WriteAndFlushEnrichedTraces`	net472	968μs	3.99μs	14.4μs	8.33	56.36 KB
#7366	`WriteAndFlushEnrichedTraces`	net6.0	701μs	729ns	2.82μs	0	41.56 KB
#7366	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	707μs	4.04μs	30.5μs	0	41.79 KB
#7366	`WriteAndFlushEnrichedTraces`	net472	903μs	3.78μs	14.6μs	8.33	56.08 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.92μs	6.88ns	26.6ns	0	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	2.74μs	7.4ns	28.7ns	0	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.82μs	2.35ns	8.48ns	0.156	0.0141	987 B
#7366	`ExecuteNonQuery`	net6.0	1.86μs	2.57ns	9.94ns	0	0	1.02 KB
#7366	`ExecuteNonQuery`	netcoreapp3.1	2.47μs	9.36ns	36.2ns	0	0	1.02 KB
#7366	`ExecuteNonQuery`	net472	2.73μs	1.53ns	5.72ns	0.149	0.0135	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.78μs	6.16ns	21.4ns	0	1.03 KB
master	`CallElasticsearch`	netcoreapp3.1	2.26μs	9.56ns	37ns	0	1.03 KB
master	`CallElasticsearch`	net472	3.56μs	4.71ns	18.2ns	0.16	1.04 KB
master	`CallElasticsearchAsync`	net6.0	1.85μs	9.52ns	45.6ns	0	1.01 KB
master	`CallElasticsearchAsync`	netcoreapp3.1	2.33μs	11.1ns	41.4ns	0	1.08 KB
master	`CallElasticsearchAsync`	net472	3.79μs	2.38ns	9.23ns	0.17	1.1 KB
#7366	`CallElasticsearch`	net6.0	1.8μs	8.68ns	33.6ns	0	1.03 KB
#7366	`CallElasticsearch`	netcoreapp3.1	2.28μs	6.15ns	23ns	0	1.03 KB
#7366	`CallElasticsearch`	net472	3.56μs	3.33ns	12.4ns	0.16	1.04 KB
#7366	`CallElasticsearchAsync`	net6.0	1.79μs	8.62ns	34.5ns	0	1.01 KB
#7366	`CallElasticsearchAsync`	netcoreapp3.1	2.38μs	7.13ns	27.6ns	0	1.08 KB
#7366	`CallElasticsearchAsync`	net472	3.81μs	3.69ns	13.8ns	0.171	1.1 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.98μs	6.54ns	25.3ns	0	952 B
master	`ExecuteAsync`	netcoreapp3.1	2.3μs	1.89ns	7.32ns	0	952 B
master	`ExecuteAsync`	net472	2.42μs	2.32ns	8.69ns	0.145	915 B
#7366	`ExecuteAsync`	net6.0	1.86μs	2.23ns	8.34ns	0	952 B
#7366	`ExecuteAsync`	netcoreapp3.1	2.37μs	6.75ns	26.1ns	0	952 B
#7366	`ExecuteAsync`	net472	2.51μs	2.07ns	8.02ns	0.138	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	6.91μs	4.85ns	17.5ns	0	2.36 KB
master	`SendAsync`	netcoreapp3.1	8.38μs	22.4ns	86.9ns	0	2.9 KB
master	`SendAsync`	net472	12.5μs	7.53ns	29.2ns	0.498	3.18 KB
#7366	`SendAsync`	net6.0	6.97μs	6.35ns	24.6ns	0	2.36 KB
#7366	`SendAsync`	netcoreapp3.1	8.44μs	24.2ns	87.3ns	0	2.9 KB
#7366	`SendAsync`	net472	12.2μs	9.37ns	36.3ns	0.487	3.18 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7366

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1	257.73 KB	274.3 KB	16.58 KB	6.43%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑netcoreapp3.1	42.64 KB	44.33 KB	1.69 KB	3.96%

Fewer allocations 🎉 in #7366

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net472	348.55 KB	286.72 KB	-61.83 KB	-17.74%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark`	net6.0	47.6μs	215ns	803ns	43.77 KB
master	`StringConcatBenchmark`	netcoreapp3.1	49.5μs	280ns	1.81μs	42.64 KB
master	`StringConcatBenchmark`	net472	57.2μs	241ns	868ns	57.34 KB
master	`StringConcatAspectBenchmark`	net6.0	465μs	1.42μs	5.11μs	258.26 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	541μs	2.5μs	10μs	257.73 KB
master	`StringConcatAspectBenchmark`	net472	407μs	2.14μs	10.9μs	348.55 KB
#7366	`StringConcatBenchmark`	net6.0	48.3μs	207ns	826ns	43.74 KB
#7366	`StringConcatBenchmark`	netcoreapp3.1	50.5μs	295ns	2.61μs	44.33 KB
#7366	`StringConcatBenchmark`	net472	57.4μs	136ns	510ns	57.34 KB
#7366	`StringConcatAspectBenchmark`	net6.0	456μs	1.9μs	6.57μs	258.04 KB
#7366	`StringConcatAspectBenchmark`	netcoreapp3.1	537μs	2.8μs	14μs	274.3 KB
#7366	`StringConcatAspectBenchmark`	net472	410μs	2.19μs	11.4μs	286.72 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.62μs	4.41ns	17.1ns	0	1.7 KB
master	`EnrichedLog`	netcoreapp3.1	3.29μs	9.02ns	35ns	0	1.7 KB
master	`EnrichedLog`	net472	4μs	4.26ns	16.5ns	0.258	1.64 KB
#7366	`EnrichedLog`	net6.0	2.59μs	11.9ns	47.7ns	0	1.7 KB
#7366	`EnrichedLog`	netcoreapp3.1	3.41μs	9.62ns	37.2ns	0	1.7 KB
#7366	`EnrichedLog`	net472	4.02μs	3.67ns	14.2ns	0.241	1.64 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`EnrichedLog`	net6.0	123μs	110ns	398ns	4.31 KB
master	`EnrichedLog`	netcoreapp3.1	128μs	203ns	787ns	4.31 KB
master	`EnrichedLog`	net472	167μs	38.1ns	148ns	4.52 KB
#7366	`EnrichedLog`	net6.0	123μs	102ns	394ns	4.31 KB
#7366	`EnrichedLog`	netcoreapp3.1	128μs	194ns	753ns	4.31 KB
#7366	`EnrichedLog`	net472	168μs	110ns	426ns	4.52 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	4.99μs	19.9ns	77.1ns	0	2.26 KB
master	`EnrichedLog`	netcoreapp3.1	6.71μs	20.4ns	78.9ns	0	2.26 KB
master	`EnrichedLog`	net472	7.46μs	9.81ns	38ns	0.3	2.08 KB
#7366	`EnrichedLog`	net6.0	4.84μs	10.6ns	41.2ns	0	2.26 KB
#7366	`EnrichedLog`	netcoreapp3.1	6.74μs	26.1ns	101ns	0	2.26 KB
#7366	`EnrichedLog`	net472	7.65μs	6.64ns	24.8ns	0.306	2.08 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.99μs	10.1ns	44.2ns	0	1.2 KB
master	`SendReceive`	netcoreapp3.1	2.71μs	13.2ns	52.8ns	0	1.2 KB
master	`SendReceive`	net472	3.12μs	7.59ns	29.4ns	0.189	1.2 KB
#7366	`SendReceive`	net6.0	1.98μs	1.78ns	6.65ns	0	1.2 KB
#7366	`SendReceive`	netcoreapp3.1	2.57μs	1.75ns	6.78ns	0	1.2 KB
#7366	`SendReceive`	net472	3.21μs	4.77ns	18.5ns	0.177	1.2 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	4.13μs	16.6ns	64.2ns	0	1.58 KB
master	`EnrichedLog`	netcoreapp3.1	5.6μs	12.3ns	47.7ns	0	1.63 KB
master	`EnrichedLog`	net472	6.65μs	11.3ns	43.9ns	0.3	2.03 KB
#7366	`EnrichedLog`	net6.0	4.26μs	2.76ns	10.3ns	0	1.58 KB
#7366	`EnrichedLog`	netcoreapp3.1	5.61μs	11.1ns	42.8ns	0	1.63 KB
#7366	`EnrichedLog`	net472	6.78μs	7.7ns	29.8ns	0.306	2.03 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	752ns	3.61ns	14ns	0	576 B
master	`StartFinishSpan`	netcoreapp3.1	916ns	4.94ns	29.6ns	0	576 B
master	`StartFinishSpan`	net472	898ns	0.0935ns	0.35ns	0.0903	578 B
master	`StartFinishScope`	net6.0	907ns	4.31ns	16.7ns	0	696 B
master	`StartFinishScope`	netcoreapp3.1	1.15μs	1.32ns	5.1ns	0	696 B
master	`StartFinishScope`	net472	1.09μs	0.242ns	0.937ns	0.104	658 B
#7366	`StartFinishSpan`	net6.0	743ns	3.54ns	21.8ns	0	576 B
#7366	`StartFinishSpan`	netcoreapp3.1	936ns	4.39ns	18.1ns	0	576 B
#7366	`StartFinishSpan`	net472	969ns	0.264ns	0.987ns	0.0912	578 B
#7366	`StartFinishScope`	net6.0	889ns	4.39ns	18.1ns	0	696 B
#7366	`StartFinishScope`	netcoreapp3.1	1.13μs	5.35ns	20.7ns	0	696 B
#7366	`StartFinishScope`	net472	1.12μs	0.31ns	1.2ns	0.101	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	1.04μs	5.36ns	25.7ns	0	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.33μs	6.48ns	29ns	0	696 B
master	`RunOnMethodBegin`	net472	1.4μs	1.54ns	5.76ns	0.105	658 B
#7366	`RunOnMethodBegin`	net6.0	1.06μs	0.852ns	3.19ns	0	696 B
#7366	`RunOnMethodBegin`	netcoreapp3.1	1.31μs	6.7ns	33.5ns	0	696 B
#7366	`RunOnMethodBegin`	net472	1.41μs	0.819ns	3.06ns	0.0988	658 B

andrewlock

I wonder what impact on startup this will have by not initializing it in the background. As best I can tell from comparing the execution benchmarks to master, this adds about 12% to startup time when debugging is not enabled, which seems problematic for many scenarios. It's not clear where all the source of that overhead is, but I suspect we could reduce some of it by not initializing most things until the debugger is actually enabled

All of the initialization seems very complex, much more so than I would have anticipated, but I'm not sure how much of that is just due to my not being familiar with the DI code in general.

tracer/src/Datadog.Trace/ClrProfiler/Instrumentation.cs

tracer/src/Datadog.Trace/Configuration/DynamicConfigurationManager.cs

andrewlock · 2025-08-15T10:40:14Z

tracer/src/Datadog.Trace/Debugger/DebuggerManager.cs

        internal string ServiceName { get; private set; }

-        private async Task<bool> WaitForDiscoveryServiceAsync(IDiscoveryService discoveryService, Task shutdownTask)
+        private string GetServiceName(TracerSettings tracerSettings)


I know it existed before, but the fact that it does is more evidence that our layering here is just wrong 😢 I still think we need to refactor this significantly to accept the fact that there's a dependence of debugger on tracing. It shouldn't be intiailizing independently like this, as it's a recipe for mismatch. All for later of course, but it's becoming clearer and clearer

andrewlock · 2025-08-15T10:42:06Z

tracer/src/Datadog.Trace/Debugger/DebuggerManager.cs

+            try
            {
-                return true;
+                return TraceUtil.NormalizeTag(tracerSettings.ServiceName ?? TracerManager.Instance.DefaultServiceName);


The TracerManager.Instance access is a big issue frankly. It could refer to a completely different TracerSettings object for example. Essentially, this DebuggerManager should be created as part of the TracerManager creation, and then it can just pass in the values you need. Again, something for a different PR that we can work on.

andrewlock · 2025-08-15T10:44:13Z

tracer/src/Datadog.Trace/Debugger/DebuggerManager.cs

+                return _processExit.Task;
+            }
+
+            OneTimeSetup(tracerSettings);


I have some suspicions that that is the source of the extra overhead we saw coming from the refactor. Essentially you're doing a bunch of one-time setup, on the startup hot path, even if the debugger is never used. I think we should try to do this work lazily (I started working on a PR to do that)

In case the symbol upload is disabled, or all 3 products (CO, ER and DI) are explicitly disabled, the symbols upload should be skipped. The thing is that if the features are not explicitly disabled, but empty, we will want to upload symbols as soon as possible even if they are not currently enabled, in case it is enabled remotely. Otherwise, the user may not see symbols until we upload them all. Note, that we don't start to upload symbols untill we got a rc request to do so.

Fixed in 79622fa
Let me know if you still want to to disable it completly untill we got the first rc request to enable it.

See also ddfb2c4

Let me know if you still want to to disable it completly untill we got the first rc request to enable it

I think we should do that, because otherwise we're doing a bunch of work (and impacting performance) that most of our customers will never benefit from. I think we should definitely be lazy for all of this setup wherever we can be, because it has a clear impact on our benchmarks, and for large apps that could end up being very significant

After bfa3f79b5c87db7e916afbdc9375dce6dfcf2ae0 symdb won't be on by default if di is explicitly disabled

won't be on by default if di is explicitly disabled

Yeah, the concern is that's basically never going to be the case, so it's effectively still always going to be on by default for customers

tracer/src/Datadog.Trace/Debugger/DebuggerSettings.cs

andrewlock · 2025-08-15T11:15:19Z

tracer/src/Datadog.Trace/Debugger/DebuggerSettings.cs

+        public bool DynamicInstrumentationEnabled
+        {
+            get =>
+                (_internalDynamicInstrumentationEnabled == true && DynamicSettings.DynamicInstrumentationEnabled == null)
+             || (_internalDynamicInstrumentationEnabled == null && DynamicSettings.DynamicInstrumentationEnabled == true);
+            init { }
+        }


I'd suggest you rewrite this to be similar to the tracer version e.g.

Suggested change

public bool DynamicInstrumentationEnabled

{

get =>

(_internalDynamicInstrumentationEnabled == true && DynamicSettings.DynamicInstrumentationEnabled == null)

|| (_internalDynamicInstrumentationEnabled == null && DynamicSettings.DynamicInstrumentationEnabled == true);

init { }

}

public bool DynamicInstrumentationEnabled

=> DynamicSettings.DynamicInstrumentationEnabled ?? _internalDynamicInstrumentationEnabled;

And then remove all the other changes above, make _internalDynamicInstrumentationEnabled a bool and do this change:

- DynamicInstrumentationEnabled = config + _internalDynamicInstrumentationEnabled = config .WithKeys(ConfigurationKeys.Debugger.DynamicInstrumentationEnabled) .AsBool(false);

But then how can I distinguish between empty/null to explicit false? I need to know that to decide if a feature might be enabled in the future or not.

I need to know that to decide if a feature might be enabled in the future or not.

Remote configuration overrides the value anyway, so it can always be added in the future I think? And if not, then we have a layering problem and confusion issue across products :/

Does it make sense now? bfa3f79b5c87db7e916afbdc9375dce6dfcf2ae0

tracer/src/Datadog.Trace/Debugger/DynamicInstrumentation.cs

tracer/src/Datadog.Trace/Debugger/ExceptionAutoInstrumentation/ExceptionTrackManager.cs

## Summary of changes Disables the debugger initialization if it's not explicitly enabled ## Reason for change We have been seeing some hangs in CI which seem to be caused by the debugger (based on log messages) and I suspect are due to [this refactoring](015d218). Rather than revert that large piece of refactoring (and causing massive conflicts for the [remote-enablement PR](#7366)), for now we primarily want to ensure customers _not_ using the debugger are not impacted. ## Implementation details Check the debugger and exception replay settings to see if the debugger is required. If not, bail out of initialization immediately. ## Test coverage This is the test

GreenMatan

submitting first batch of comments, will keep reviewing and potentially send another batch.
Nice work!

tracer/src/Datadog.Trace/ClrProfiler/Instrumentation.cs

GreenMatan · 2025-09-02T11:53:55Z

tracer/src/Datadog.Trace/ClrProfiler/Instrumentation.cs

            else
            {
-                Log.Information($"Dynamic Instrumentation is disabled. To enable it, please set {ConfigurationKeys.Debugger.DynamicInstrumentationEnabled} environment variable to 'true'.");
+                if (!manager.DebuggerSettings.DynamicInstrumentationEnabled)


#7366 (comment)

tracer/src/Datadog.Trace/Configuration/DynamicConfigurationManager.cs

GreenMatan · 2025-09-02T11:54:44Z

tracer/src/Datadog.Trace/Configuration/DynamicConfigurationManager.cs

+                /*
+                if (debugLogsEnabled != null && debugLogsEnabled.Value != GlobalSettings.Instance.DebugEnabled)
+                {
+                    GlobalSettings.SetDebugEnabledInternal(debugLogsEnabled.Value);
+                    Security.Instance.SetDebugEnabled(debugLogsEnabled.Value);

-            Log.Information("Applying new dynamic configuration");
+                    NativeMethods.UpdateSettings(new[] { ConfigurationKeys.DebugEnabled }, new[] { debugLogsEnabled.Value ? "1" : "0" });
+                }
+                */


do we need this comment or it's a left over?

It was there before and I'm not sure if should I delete it or not

GreenMatan · 2025-09-02T12:02:36Z

tracer/test/Datadog.Trace.Tests/Debugger/DynamicInstrumentationTests.cs

        var debugger = new DynamicInstrumentation(settings, discoveryService, rcmSubscriptionManagerMock, lineProbeResolver, snapshotUploader, diagnosticsUploader, probeStatusPoller, updater, new DogStatsd.NoOpStatsd());
        debugger.Initialize();
-
+        await Task.Delay(1000);


why do we need this delay?

To let the DI background initialization to run theoritically so in case we have a bug in our enabling condition this test will fail.

We should not have any Task.Delay in tests, because it will make them flaky. I practically guarantee that this will take longer than 1s in some cases, because the CI machines run at basically 100% capacity.

I strongly suggest having a deterministic way to know whether initialization has completed. If you can't, that suggests we need to redesign the code - APIs should be testable by default (e.g. no static access too 😉)

please let's avoid using any Thread.Sleep or Task.Delay in tests.

Fixed in 5683a7c78bfbca902bc8eb61710df96ff20c78ad

I strongly suggest having a deterministic way to know whether initialization has completed

We have a way for that, but in this case we need the opposite. Anyway I prefer to leave it as is was before, we don't have to change it now.

GreenMatan · 2025-09-02T12:07:39Z

tracer/src/Datadog.Trace/Debugger/DebuggerManager.cs

+                DynamicInstrumentation? instance = null;
+                var created = false;
+                lock (_syncLock)
+                {
+                    if (DynamicInstrumentation == null)
                    {
-                        // ignore ObjectDisposedException or SemaphoreFullException
+                        _dynamicInstrumentation = DebuggerFactory.CreateDynamicInstrumentation(
+                            discoveryService,
+                            RcmSubscriptionManager.Instance,
+                            tracerSettings,
+                            ServiceName,
+                            debuggerSettings,
+                            tracerManager.GitMetadataTagsProvider);
+                        created = true;
                    }
+
+                    instance = _dynamicInstrumentation;
+                }
+
+                if (created && instance != null)
+                {
+                    Log.Debug("Dynamic Instrumentation has been created");
+                    instance.Initialize();
+                    tracerManager.Telemetry.ProductChanged(TelemetryProductType.DynamicInstrumentation, enabled: true, error: null);
                }


since you lock only specific part, do we care if we end up calling instance.Initialize on DynamicInstrumentation instance while another thread tries to create/destroy it potentially? Do we have potential race here?

Fixed in 1d6bfc7

andrewlock

Thanks, I left a few comments, in particular I think we need to try to reduce the impact when the debugger is not enabled.

That said, I think the main thing missing at this point is integration tests for the remote enablement (unless they're already there and I just missed them!). I think we should have tests for:

start with the debugger enabled, disable it, then re-enable it
start with the debugger disabled, enable it, then re-disable it

I believe there are some ASM tests that do something similar, so might be worth looking at those for inspiration! Thanks

tracer/src/Datadog.Trace/ClrProfiler/Instrumentation.cs

andrewlock · 2025-09-03T09:09:24Z

tracer/src/Datadog.Trace/Debugger/DebuggerManager.cs

+        private readonly TaskCompletionSource<bool> _processExit;
+        private volatile bool _isDebuggerEndpointAvailable;
        private int _initialized;
+        private CancellationTokenSource? _diDebounceCts;


We shouldn't use CancellationTokenSource in paths that can be hit during shutdown, because it can cause crashes. We generally have to use TaskCompletionSource instead (even though it's a bit of a pain)

Thanks for pointing that. Fixed in f3d6f4c949964231419ab232c0b5939abb08ac8e

andrewlock · 2025-09-03T09:13:14Z

tracer/src/Datadog.Trace/Debugger/DebuggerManager.cs

+                return _processExit.Task;
+            }
+
+            OneTimeSetup(tracerSettings);


Let me know if you still want to to disable it completly untill we got the first rc request to enable it

I think we should do that, because otherwise we're doing a bunch of work (and impacting performance) that most of our customers will never benefit from. I think we should definitely be lazy for all of this setup wherever we can be, because it has a clear impact on our benchmarks, and for large apps that could end up being very significant

andrewlock · 2025-09-03T09:17:58Z

tracer/src/Datadog.Trace/Debugger/DebuggerManager.cs

+                    writer.WriteEndObject();
                }

                Log.Information("DATADOG DEBUGGER CONFIGURATION - {Configuration}", stringWriter.ToString());


Given you always write this log (which I think is fine, as it mirrors the tracer version 👍), you can probably change the other logs that are used by tests to use this one, instead of writing additional logs

I think that we use today only this initialization log in tests (there ar emore but not related to initialization)
Dynamic Instrumentation is disabled. To enable it, please set DD_DYNAMIC_INSTRUMENTATION_ENABLED environment variable to 'true'.
We can change the test to use the configuration value but we still want this log (without this log).

andrewlock · 2025-09-03T09:18:46Z

tracer/src/Datadog.Trace/Debugger/DynamicInstrumentation.cs

-            if (originalState != 0)
+            if (!_settings.DynamicInstrumentationEnabled)
            {
+                Log.Information("Dynamic Instrumentation is disabled. To enable it, please set DD_DYNAMIC_INSTRUMENTATION_ENABLED environment variable to 'true'.");


nit: You're already writing this log elsewhere, we should remove one of them (this is a problem today, we write the same log twice)

Fixed in 9cc8537e8fce4bd242b9c5798bfc51c00d140e38

andrewlock · 2025-09-03T09:19:27Z

tracer/src/Datadog.Trace/Debugger/DynamicInstrumentation.cs

-                }
-
-                Log.Information("Dynamic Instrumentation initialization started");
+            Log.Information("Dynamic Instrumentation initialization started");


nit: this should probably be debug

Fixed in 9cc8537e8fce4bd242b9c5798bfc51c00d140e38

andrewlock · 2025-09-03T09:22:22Z

tracer/src/Datadog.Trace/Debugger/DynamicInstrumentation.cs

-                Interlocked.Exchange(ref _initState, 0);
-            }
+            // Start initialization in background
+            _ = Task.Run(InitializeAsync)


There's so much "in the background" stuff that I get lost 😢 Given everything is started "in the background" anyway in Instrumentation, do we really need to make everything explicitly use Task.Run? It would be much easier to follow if the DebuggerManager for example, could just do something like this:

Task allTasks = Task.WhenAll( DynamicInstrumentation.InitializeAsync(), SymbolUploader.InitializeAsync(), SomethingElse.InitializeAsync());

Not required, and could be another PR if you want, but I think it would make a lot of these things much easier to follow

Deleted some Task.Run in d3b96db

andrewlock · 2025-09-03T09:33:18Z

tracer/src/Datadog.Trace/Debugger/ExceptionAutoInstrumentation/ExceptionReplaySettings.cs

+            Enabled = config.WithKeys(ConfigurationKeys.Debugger.ExceptionReplayEnabled).AsBool(false);
+
+            _internalExceptionReplayEnabled = config.WithKeys(ConfigurationKeys.Debugger.ExceptionReplayEnabled).AsBool();


You are reading the same setting from all of the sources, multiple times, this is expensive, and shouldn't be necessary.

If you need to do this (I'm still not entirely convinced you do - you should always be able to enable via remote config) you can do it like this:

Suggested change

Enabled = config.WithKeys(ConfigurationKeys.Debugger.ExceptionReplayEnabled).AsBool(false);

_internalExceptionReplayEnabled = config.WithKeys(ConfigurationKeys.Debugger.ExceptionReplayEnabled).AsBool();

var enabledResult = config.WithKeys(ConfigurationKeys.Debugger.ExceptionReplayEnabled).AsBool(false);

Enabled = enabledResult.WithDefault(false);

CanBeEnabled = !(enabledResult.ConfigurationResult.IsValid && enabledResult.ConfigurationResult.Result == false);

Note that I changed the name of the property (and removed the unecessary field), because it's currently confusing to have ExceptionReplaySettings.Enabled and IsExceptionReplayDisabled meaning different things

Does it make sense now? bfa3f79b5c87db7e916afbdc9375dce6dfcf2ae0

andrewlock · 2025-09-03T09:34:00Z

tracer/src/Datadog.Trace/Debugger/ExceptionAutoInstrumentation/ExceptionTrackManager.cs

            _exceptionProcessorTask = Task.Factory.StartNew(
-                                               async () => await StartExceptionProcessingAsync(_cts.Token).ConfigureAwait(false), TaskCreationOptions.LongRunning)
+                                               () => StartExceptionProcessingAsync(),
+                                               TaskCreationOptions.LongRunning)


You probably need to make sure you're using the default scheduler if you're using StartNew instead of Task.Run()

Fixed in dd64f6e89a16c1969abd9c1deb54d70d0f8c89c4

tonyredondo · 2025-09-03T10:57:39Z

tracer/src/Datadog.Trace/Debugger/DebuggerManager.cs

+                _ = Task.Run(
+                    async () =>
+                    {
+                        try
+                        {
+                            await SymbolsUploader.StartFlushingAsync().ConfigureAwait(false);
+                        }
+                        catch (Exception ex)
+                        {
+                            Log.Error(ex, "Failed to initialize symbol uploader");
+                        }
+                    });


Why not

.ContinueWith( t => Log.Error(t?.Exception, "Failed to initialize symbol uploader"), TaskContinuationOptions.OnlyOnFaulted);

Fixed in dd64f6e89a16c1969abd9c1deb54d70d0f8c89c4

tonyredondo · 2025-09-03T11:06:50Z

tracer/src/Datadog.Trace/Debugger/ExceptionAutoInstrumentation/ExceptionTrackManager.cs

            _exceptionProcessorTask = Task.Factory.StartNew(
-                                               async () => await StartExceptionProcessingAsync(_cts.Token).ConfigureAwait(false), TaskCreationOptions.LongRunning)
+                                               () => StartExceptionProcessingAsync(),
+                                               TaskCreationOptions.LongRunning)


This is a LongRunning task, a new thread is created for this one, but in StartExceptionProcessingAsync you have a .ConfigureAwait(false) so this new thread will be lost at that first Task.WhenAny, because on return the continuation will be schedule to a thread pool thread.

Thanks for pointing that. Fixed in dd64f6e89a16c1969abd9c1deb54d70d0f8c89c4

andrewlock · 2025-09-04T10:16:26Z

tracer/src/Datadog.Trace/ClrProfiler/Instrumentation.cs

+            bool coDisabled = !debuggerSettings.CodeOriginForSpansCanBeEnabled;
+            bool erDisabled = !manager.ExceptionReplaySettings.CanBeEnabled || debuggerSettings.DynamicSettings.ExceptionReplayEnabled == false;
+
+            if (diDisabled && coDisabled && erDisabled)


Just FYI, my concern is that these will almost never all be set (because customers would have to explicitly disable all of them) so this effectively means we will be initializing for all customers - we'll need to check the execution benchmarks to check that this doesn't have an adverse impact in general 🙂

That true but we also check later that SymDb is true which will be true only if DI is not false so even if the customer set only DI to false, we will not upload symbols.

Yeah, the concern is that's basically never going to be the case, so it's effectively still always going to be on by default for customers

Ok, I undrstand. So let's do the opposite, during startap enable initialize DebuggerManager only if one of the products is enabled (== true)

Fixed in f65a260d976c7b380e2e166f6cb689018d5b6cf4

andrewlock · 2025-09-04T10:17:21Z

tracer/src/Datadog.Trace/Debugger/DebuggerManager.cs

+                return _processExit.Task;
+            }
+
+            OneTimeSetup(tracerSettings);


won't be on by default if di is explicitly disabled

Yeah, the concern is that's basically never going to be the case, so it's effectively still always going to be on by default for customers

andrewlock · 2025-09-04T10:19:41Z

tracer/src/Datadog.Trace/Debugger/DebuggerSettings.cs

+            SymbolDatabaseUploadEnabled = config.WithKeys(ConfigurationKeys.Debugger.SymbolDatabaseUploadEnabled).AsBool(true)
+                                       && _internalDynamicInstrumentationEnabled != false;


It's important that the default value reflects the actual value, so we shouldn't manipulate the value afterwards. e.g. the following is equivalent, but records the "real" default value

Suggested change

SymbolDatabaseUploadEnabled = config.WithKeys(ConfigurationKeys.Debugger.SymbolDatabaseUploadEnabled).AsBool(true)

&& _internalDynamicInstrumentationEnabled != false;

SymbolDatabaseUploadEnabled = config.WithKeys(ConfigurationKeys.Debugger.SymbolDatabaseUploadEnabled).AsBool(_internalDynamicInstrumentationEnabled != false);

Fixed in 256bd507d924cc973d55d3bb5467e6c7e5570cf3

andrewlock · 2025-09-04T10:21:38Z

tracer/src/Datadog.Trace/Debugger/DebuggerSettings.cs

-        }
+        public bool CodeOriginForSpansEnabled => _internalCodeOriginForSpansEnabled == true || DynamicSettings.CodeOriginEnabled == true;
+
+        public bool CodeOriginForSpansCanBeEnabled => _internalCodeOriginForSpansEnabled != false && DynamicSettings.CodeOriginEnabled != false;


I don't think "can be enabled" should be controlled by dynamic config? 🤔:

Suggested change

public bool CodeOriginForSpansCanBeEnabled => _internalCodeOriginForSpansEnabled != false && DynamicSettings.CodeOriginEnabled != false;

public bool CodeOriginForSpansCanBeEnabled => _internalCodeOriginForSpansEnabled != false;

Fixed in 256bd507d924cc973d55d3bb5467e6c7e5570cf3

andrewlock · 2025-09-04T10:22:08Z

tracer/src/Datadog.Trace/Debugger/ExceptionAutoInstrumentation/ExceptionReplaySettings.cs

+            var enabledResult = config.WithKeys(ConfigurationKeys.Debugger.ExceptionReplayEnabled).AsBool();
+            Enabled = enabledResult ?? false;
+            CanBeEnabled = enabledResult != false;


This doesn't record the correct value, so you need to use AsBoolResult() and then call WithDefault() etc afterwards

Fixed in 256bd507d924cc973d55d3bb5467e6c7e5570cf3

…nly when di might be enabled

…code origin tag (CO is on when DI is on)

This reverts commit b7586c6.

…gerManagerTests.cs

## Summary of changes This [PR ](#7366) breaks ER system tests because it ommits an obosolete environment variable key, this PR fix that by returning that key.

…nt (#7366)" This reverts commit bfd542f.

## Summary of changes This PR finalizes the work introduced in [Debugger In-Product Enablement](#7366) and [RC Multi-Config Support](#7536) by integrating the new debugger dynamic configuration keys into both the dynamic configuration source and the APM tracing merger. ## Test coverage DebuggerManagerDynamicTests

github-actions bot added area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) area:debugger labels Aug 7, 2025

This comment has been minimized.

Sign in to view

dudikeleti force-pushed the dudik/debugger-remote-enablement branch 2 times, most recently from a971687 to e22ff90 Compare August 10, 2025 00:14

dudikeleti marked this pull request as ready for review August 11, 2025 09:40

dudikeleti requested review from a team as code owners August 11, 2025 09:40

dudikeleti force-pushed the dudik/debugger-remote-enablement branch from e22ff90 to eee68bd Compare August 13, 2025 00:46

dudikeleti requested a review from a team as a code owner August 13, 2025 00:46

dudikeleti force-pushed the dudik/debugger-remote-enablement branch from 58df774 to ac5d278 Compare August 13, 2025 13:31

andrewlock reviewed Aug 15, 2025

View reviewed changes

andrewlock mentioned this pull request Aug 18, 2025

Disable debugger initialization if it's not enabled initially #7398

Merged

dudikeleti force-pushed the dudik/debugger-remote-enablement branch from 1782ee2 to 41e5d62 Compare September 1, 2025 07:06

dudikeleti requested review from a team as code owners September 1, 2025 07:06

GreenMatan approved these changes Sep 2, 2025

View reviewed changes

dudikeleti force-pushed the dudik/debugger-remote-enablement branch from ddfb2c4 to 1d6bfc7 Compare September 2, 2025 23:08

andrewlock reviewed Sep 3, 2025

View reviewed changes

tonyredondo reviewed Sep 3, 2025

View reviewed changes

dudikeleti force-pushed the dudik/debugger-remote-enablement branch 2 times, most recently from d3b96db to 310fcc4 Compare September 4, 2025 10:08

andrewlock reviewed Sep 4, 2025

View reviewed changes

dudikeleti force-pushed the dudik/debugger-remote-enablement branch 2 times, most recently from e317a6d to 480ab96 Compare September 16, 2025 14:05

dudikeleti force-pushed the dudik/debugger-remote-enablement branch from 480ab96 to 96c7641 Compare September 24, 2025 08:08

dudikeleti added 19 commits September 27, 2025 04:34

Don't use LongRunning and replace try\catch with ContinueWith

0b354eb

Use TaskCompletionSource insteac of CancellationTokenSource

40dba26

reduce logs

169f118

Delete unneeded Task.Run

e7a649d

fix build

6ad39ad

improve how we handle debugger settings and make symdb default true o…

b9d83a2

…nly when di might be enabled

Initialize at startup only when at least one product is enabled

3b01a27

Remove private fields from settings and use Enabled & CanBeEnabled

4242120

clean logs

0c0211e

add tests

9e4dba0

Fix code origin enabled logic

90e1e81

fix compilation

9809a3e

Fix compilation

5acada6

Initialize SymbolUploader only if DI is explicitly enabled

f9d6b23

Do not skip on .net8 x86

5a773fc

Update span decoration and dynamic span integration tests to include …

d99b07b

…code origin tag (CO is on when DI is on)

Revert "Do not skip on .net8 x86"

f9a87f6

This reverts commit b7586c6.

Delete DynamicInstrumentationTests since all covered in the new Debug…

d7957ca

…gerManagerTests.cs

Skip memory assertion when there is a timeout

0808bc4

dudikeleti force-pushed the dudik/debugger-remote-enablement branch from cdbc59a to 0808bc4 Compare September 27, 2025 03:46

andrewlock approved these changes Sep 27, 2025

View reviewed changes

Skip x86 .net 8 or greater, investigate in another pr

cb3e1da

dudikeleti merged commit bfd542f into master Sep 29, 2025
153 of 156 checks passed

dudikeleti deleted the dudik/debugger-remote-enablement branch September 29, 2025 07:47

github-actions bot added this to the vNext-v3 milestone Sep 29, 2025

dudikeleti mentioned this pull request Sep 29, 2025

[ER]- Fix ER system tests #7576

Merged

dudikeleti added a commit that referenced this pull request Sep 29, 2025

[ER]- Fix ER system tests (#7576)

cf90183

## Summary of changes This [PR ](#7366) breaks ER system tests because it ommits an obosolete environment variable key, this PR fix that by returning that key.

NachoEchevarria added a commit that referenced this pull request Sep 30, 2025

Revert "[Debugger Default-On] DEBUG-3322 Debugger in-product enableme…

11a7821

…nt (#7366)" This reverts commit bfd542f.

dudikeleti mentioned this pull request Oct 1, 2025

[Debugger Default-On] Add debuger dynamic configurations keys #7595

Merged

		Enabled = config.WithKeys(ConfigurationKeys.Debugger.ExceptionReplayEnabled).AsBool(false);

		_internalExceptionReplayEnabled = config.WithKeys(ConfigurationKeys.Debugger.ExceptionReplayEnabled).AsBool();

-            Enabled = config.WithKeys(ConfigurationKeys.Debugger.ExceptionReplayEnabled).AsBool(false);
-            _internalExceptionReplayEnabled = config.WithKeys(ConfigurationKeys.Debugger.ExceptionReplayEnabled).AsBool();
+            var enabledResult = config.WithKeys(ConfigurationKeys.Debugger.ExceptionReplayEnabled).AsBool(false);
+            Enabled = enabledResult.WithDefault(false);
+            CanBeEnabled = !(enabledResult.ConfigurationResult.IsValid && enabledResult.ConfigurationResult.Result == false);

		SymbolDatabaseUploadEnabled = config.WithKeys(ConfigurationKeys.Debugger.SymbolDatabaseUploadEnabled).AsBool(true)
		&& _internalDynamicInstrumentationEnabled != false;

	public bool CodeOriginForSpansCanBeEnabled => _internalCodeOriginForSpansEnabled != false && DynamicSettings.CodeOriginEnabled != false;
	public bool CodeOriginForSpansCanBeEnabled => _internalCodeOriginForSpansEnabled != false;

[Debugger Default-On] DEBUG-3322 Debugger in-product enablement #7366

[Debugger Default-On] DEBUG-3322 Debugger in-product enablement #7366

Uh oh!

Conversation

dudikeleti commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Reason for change

Implementation details

Test coverage

Uh oh!

This comment has been minimized.

dd-trace-dotnet-ci-bot bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Execution-Time Benchmarks Report ⏱️

Uh oh!

pr-commenter bot commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #7366

Raw results

Raw results

More allocations ⚠️ in #7366

Fewer allocations 🎉 in #7366

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

More allocations ⚠️ in #7366

Fewer allocations 🎉 in #7366

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Uh oh!

andrewlock left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dudikeleti Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dudikeleti Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dudikeleti Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

dudikeleti commented Aug 7, 2025 •

edited

Loading

dd-trace-dotnet-ci-bot bot commented Aug 7, 2025 •

edited

Loading

pr-commenter bot commented Aug 8, 2025 •

edited

Loading

dudikeleti Aug 27, 2025 •

edited

Loading

dudikeleti Sep 1, 2025 •

edited

Loading

dudikeleti Sep 1, 2025 •

edited

Loading

GreenMatan Sep 2, 2025 •

edited

Loading

dudikeleti Sep 2, 2025 •

edited

Loading

dudikeleti Sep 3, 2025 •

edited

Loading