Add AWS Lamda testing for .NET 9 #6779

andrewlock · 2025-03-19T15:33:54Z

Summary of changes

Add testing for .NET 9

Reason for change

We don't currently test on .NET 9... we should 😄

Implementation details

Added an extra dimension

Test coverage

A bit more now

Other details

I have a sneaking suspicion we don't currently test on ARM64, and we obviously should...

datadog-datadog-prod-us1 · 2025-03-19T15:56:03Z

Datadog Report

Branch report: andrew/test-lambda-with-dotnet9
Commit report: 0e9a9e9
Test service: dd-trace-dotnet

✅ 0 Failed, 248980 Passed, 2392 Skipped, 20h 48m 46.45s Total Time
⌛ 6 Performance Regressions

⌛ Performance Regressions vs Default Branch (6)

This report shows up to 5 performance regressions.

Baseline - Samples.FakeDbCommand.windows.net462.json.scenarios 79.35ms (+9.67ms, +14%) - Details
CallTarget+Inlining+NGEN - Samples.FakeDbCommand.windows.net462.json.scenarios 1.08s (+77.95ms, +8%) - Details
CallTarget+Inlining+NGEN - Samples.FakeDbCommand.windows.netcoreapp31.json.scenarios 748.16ms (+60.5ms, +9%) - Details
CallTarget+Inlining+NGEN - Samples.FakeDbCommand.windows.net60.json.scenarios 703.95ms (+60.51ms, +9%) - Details
Baseline - Samples.FakeDbCommand.windows.netcoreapp31.json.scenarios 116.71ms (+13.85ms, +13%) - Details

andrewlock · 2025-03-19T16:11:27Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6779) - mean (79ms)  : 74, 85
     .   : milestone, 79,
    master - mean (69ms)  : 65, 74
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6779) - mean (1,085ms)  : crit, 1050, 1120
     .   : crit, milestone, 1085,
    master - mean (1,002ms)  : 977, 1026
     .   : milestone, 1002,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6779) - mean (117ms)  : 110, 123
     .   : milestone, 117,
    master - mean (102ms)  : 100, 104
     .   : milestone, 102,

    section CallTarget+Inlining+NGEN
    This PR (6779) - mean (748ms)  : crit, 725, 771
     .   : crit, milestone, 748,
    master - mean (686ms)  : 665, 706
     .   : milestone, 686,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6779) - mean (101ms)  : 94, 107
     .   : milestone, 101,
    master - mean (89ms)  : 87, 91
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (6779) - mean (704ms)  : crit, 685, 722
     .   : crit, milestone, 704,
    master - mean (642ms)  : 626, 659
     .   : milestone, 642,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6779) - mean (191ms)  : 185, 196
     .   : milestone, 191,
    master - mean (190ms)  : 186, 195
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (6779) - mean (1,111ms)  : 1080, 1143
     .   : milestone, 1111,
    master - mean (1,109ms)  : 1078, 1139
     .   : milestone, 1109,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6779) - mean (270ms)  : 266, 273
     .   : milestone, 270,
    master - mean (269ms)  : 265, 274
     .   : milestone, 269,

    section CallTarget+Inlining+NGEN
    This PR (6779) - mean (877ms)  : 842, 913
     .   : milestone, 877,
    master - mean (876ms)  : 847, 904
     .   : milestone, 876,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6779) - mean (262ms)  : 259, 266
     .   : milestone, 262,
    master - mean (262ms)  : 258, 265
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (6779) - mean (862ms)  : 829, 896
     .   : milestone, 862,
    master - mean (861ms)  : 824, 898
     .   : milestone, 861,

andrewlock · 2025-03-19T16:41:25Z

Benchmarks Report for tracer 🐌

Benchmarks for #6779 compared to master:

1 benchmarks are faster, with geometric mean 1.156
1 benchmarks are slower, with geometric mean 1.135
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.91μs	42.9ns	231ns	0.0124	0.00412	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.3μs	55.4ns	323ns	0.0243	0.0097	0	5.81 KB
master	`StartStopWithChild`	net472	16.1μs	44.3ns	166ns	1.04	0.304	0.0962	6.22 KB
#6779	`StartStopWithChild`	net6.0	8.02μs	43.8ns	263ns	0.0193	0.00772	0	5.61 KB
#6779	`StartStopWithChild`	netcoreapp3.1	9.93μs	55ns	335ns	0.0148	0.00492	0	5.81 KB
#6779	`StartStopWithChild`	net472	16.2μs	32.5ns	117ns	1.03	0.296	0.088	6.2 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	517μs	280ns	969ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	658μs	407ns	1.52μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	844μs	471ns	1.83μs	0.422	3.3 KB
#6779	`WriteAndFlushEnrichedTraces`	net6.0	494μs	523ns	2.03μs	0	2.7 KB
#6779	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	662μs	597ns	2.23μs	0	2.7 KB
#6779	`WriteAndFlushEnrichedTraces`	net472	880μs	441ns	1.65μs	0.428	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	129μs	565ns	2.19μs	0.194	14.47 KB
master	`SendRequest`	netcoreapp3.1	148μs	131ns	490ns	0.223	17.27 KB
master	`SendRequest`	net472	0.000427ns	0.000245ns	0.000918ns	0	0 b
#6779	`SendRequest`	net6.0	129μs	557ns	2.16μs	0.193	14.47 KB
#6779	`SendRequest`	netcoreapp3.1	146μs	299ns	1.16μs	0.218	17.27 KB
#6779	`SendRequest`	net472	0.000383ns	0.000162ns	0.000607ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	597μs	3.88μs	37.8μs	0.604	0	0	41.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	682μs	2.83μs	10.2μs	0.332	0	0	41.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	851μs	4.11μs	15.9μs	8.08	2.55	0.425	53.32 KB
#6779	`WriteAndFlushEnrichedTraces`	net6.0	579μs	2.99μs	19.8μs	0.622	0	0	41.57 KB
#6779	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	703μs	4μs	32μs	0.357	0	0	41.66 KB
#6779	`WriteAndFlushEnrichedTraces`	net472	860μs	4.42μs	19.8μs	8.25	2.6	0.434	53.26 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.34μs	1.54ns	5.77ns	0.0142	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.79μs	1.82ns	7.06ns	0.0136	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.06μs	2.46ns	9.52ns	0.156	0.00103	987 B
#6779	`ExecuteNonQuery`	net6.0	1.25μs	1.17ns	4.51ns	0.0142	0	1.02 KB
#6779	`ExecuteNonQuery`	netcoreapp3.1	1.77μs	2.28ns	8.84ns	0.0133	0	1.02 KB
#6779	`ExecuteNonQuery`	net472	2.07μs	2.61ns	10.1ns	0.156	0.00103	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.24μs	0.25ns	0.936ns	0.0136	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.59μs	2.22ns	8.32ns	0.0135	976 B
master	`CallElasticsearch`	net472	2.71μs	1.55ns	5.81ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.32μs	0.547ns	2.05ns	0.0132	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.63μs	0.495ns	1.92ns	0.0143	1.02 KB
master	`CallElasticsearchAsync`	net472	2.71μs	1.18ns	4.41ns	0.167	1.05 KB
#6779	`CallElasticsearch`	net6.0	1.27μs	1.55ns	6.02ns	0.0134	976 B
#6779	`CallElasticsearch`	netcoreapp3.1	1.62μs	1.59ns	6.15ns	0.0131	976 B
#6779	`CallElasticsearch`	net472	2.59μs	1.6ns	6ns	0.157	995 B
#6779	`CallElasticsearchAsync`	net6.0	1.27μs	0.661ns	2.47ns	0.0133	952 B
#6779	`CallElasticsearchAsync`	netcoreapp3.1	1.63μs	2.16ns	8.38ns	0.014	1.02 KB
#6779	`CallElasticsearchAsync`	net472	2.76μs	1.23ns	4.59ns	0.167	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.32μs	0.707ns	2.64ns	0.0131	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.63μs	2.42ns	9.04ns	0.0128	952 B
master	`ExecuteAsync`	net472	1.78μs	0.353ns	1.32ns	0.145	915 B
#6779	`ExecuteAsync`	net6.0	1.29μs	0.426ns	1.59ns	0.0129	952 B
#6779	`ExecuteAsync`	netcoreapp3.1	1.74μs	0.796ns	2.98ns	0.0122	952 B
#6779	`ExecuteAsync`	net472	1.73μs	0.685ns	2.47ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.43μs	1.83ns	6.86ns	0.031	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.36μs	1.44ns	5.57ns	0.0375	2.85 KB
master	`SendAsync`	net472	7.55μs	2.14ns	8ns	0.496	3.12 KB
#6779	`SendAsync`	net6.0	4.34μs	1.65ns	6.17ns	0.0327	2.31 KB
#6779	`SendAsync`	netcoreapp3.1	5.38μs	2.71ns	10.1ns	0.0376	2.85 KB
#6779	`SendAsync`	net472	7.6μs	21.3ns	76.9ns	0.492	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.62μs	1.01ns	3.78ns	0.0238	1.69 KB
master	`EnrichedLog`	netcoreapp3.1	2.35μs	1.11ns	4.29ns	0.0224	1.69 KB
master	`EnrichedLog`	net472	2.57μs	1.6ns	6ns	0.257	1.62 KB
#6779	`EnrichedLog`	net6.0	1.57μs	3.69ns	14.3ns	0.0236	1.69 KB
#6779	`EnrichedLog`	netcoreapp3.1	2.3μs	1.65ns	6.38ns	0.023	1.69 KB
#6779	`EnrichedLog`	net472	2.48μs	1.2ns	4.48ns	0.258	1.62 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	111μs	80.5ns	301ns	0.0557	0	4.3 KB
master	`EnrichedLog`	netcoreapp3.1	117μs	157ns	607ns	0.0581	0	4.3 KB
master	`EnrichedLog`	net472	152μs	157ns	609ns	0.685	0.228	4.49 KB
#6779	`EnrichedLog`	net6.0	114μs	116ns	433ns	0.0564	0	4.3 KB
#6779	`EnrichedLog`	netcoreapp3.1	116μs	116ns	403ns	0.0579	0	4.3 KB
#6779	`EnrichedLog`	net472	150μs	122ns	455ns	0.677	0.226	4.49 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.96μs	1.47ns	5.51ns	0.0312	2.25 KB
master	`EnrichedLog`	netcoreapp3.1	4.17μs	4.88ns	18.9ns	0.0289	2.25 KB
master	`EnrichedLog`	net472	4.76μs	2.5ns	9.69ns	0.328	2.07 KB
#6779	`EnrichedLog`	net6.0	3.05μs	0.942ns	3.65ns	0.0321	2.25 KB
#6779	`EnrichedLog`	netcoreapp3.1	4.28μs	2.38ns	9.21ns	0.0299	2.25 KB
#6779	`EnrichedLog`	net472	4.86μs	3.6ns	14ns	0.329	2.07 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.31μs	1.43ns	5.14ns	0.0163	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.85μs	0.713ns	2.67ns	0.0158	1.14 KB
master	`SendReceive`	net472	2.06μs	0.874ns	3.27ns	0.183	1.16 KB
#6779	`SendReceive`	net6.0	1.31μs	0.651ns	2.44ns	0.0164	1.14 KB
#6779	`SendReceive`	netcoreapp3.1	1.79μs	1.02ns	3.83ns	0.0151	1.14 KB
#6779	`SendReceive`	net472	2.07μs	1.16ns	4.48ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.85μs	1.13ns	4.07ns	0.0227	1.62 KB
master	`EnrichedLog`	netcoreapp3.1	3.83μs	0.963ns	3.73ns	0.023	1.67 KB
master	`EnrichedLog`	net472	4.37μs	2.49ns	9.31ns	0.328	2.07 KB
#6779	`EnrichedLog`	net6.0	2.73μs	0.994ns	3.72ns	0.022	1.62 KB
#6779	`EnrichedLog`	netcoreapp3.1	4.02μs	3.52ns	13.2ns	0.0218	1.67 KB
#6779	`EnrichedLog`	net472	4.45μs	4.12ns	15.9ns	0.327	2.07 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6779

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.156	554.09	479.33

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	396ns	0.121ns	0.471ns	0.0081	576 B
master	`StartFinishSpan`	netcoreapp3.1	592ns	0.366ns	1.37ns	0.00774	576 B
master	`StartFinishSpan`	net472	594ns	0.319ns	1.23ns	0.0916	578 B
master	`StartFinishScope`	net6.0	554ns	0.151ns	0.523ns	0.00986	696 B
master	`StartFinishScope`	netcoreapp3.1	722ns	0.372ns	1.39ns	0.00937	696 B
master	`StartFinishScope`	net472	896ns	0.264ns	1.02ns	0.105	658 B
#6779	`StartFinishSpan`	net6.0	434ns	0.443ns	1.72ns	0.00803	576 B
#6779	`StartFinishSpan`	netcoreapp3.1	581ns	0.734ns	2.84ns	0.00785	576 B
#6779	`StartFinishSpan`	net472	638ns	0.132ns	0.495ns	0.0916	578 B
#6779	`StartFinishScope`	net6.0	479ns	0.134ns	0.519ns	0.0098	696 B
#6779	`StartFinishScope`	netcoreapp3.1	764ns	0.465ns	1.8ns	0.00939	696 B
#6779	`StartFinishScope`	net472	843ns	0.14ns	0.505ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6779

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.135	594.80	674.96

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	595ns	0.146ns	0.526ns	0.00989	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	938ns	0.635ns	2.46ns	0.00918	696 B
master	`RunOnMethodBegin`	net472	1.07μs	0.232ns	0.898ns	0.104	658 B
#6779	`RunOnMethodBegin`	net6.0	675ns	0.158ns	0.592ns	0.00988	696 B
#6779	`RunOnMethodBegin`	netcoreapp3.1	886ns	0.447ns	1.73ns	0.00936	696 B
#6779	`RunOnMethodBegin`	net472	1.11μs	0.17ns	0.66ns	0.104	658 B

## Summary of changes Add testing for .NET 9 ## Reason for change We don't currently test on .NET 9... we should 😄 ## Implementation details Added an extra dimension ## Test coverage A bit more now ## Other details I have a sneaking suspicion we _don't_ currently test on ARM64, and we obviously should...

Add serverless testing for .NET 9

0e9a9e9

andrewlock added area:tests unit tests, integration tests area:serverless labels Mar 19, 2025

andrewlock requested a review from a team as a code owner March 19, 2025 15:33

github-actions bot added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Mar 19, 2025

lucaspimentel approved these changes Mar 19, 2025

View reviewed changes

andrewlock merged commit f1eddc0 into master Mar 19, 2025
127 of 130 checks passed

andrewlock deleted the andrew/test-lambda-with-dotnet9 branch March 19, 2025 17:35

github-actions bot added this to the vNext-v3 milestone Mar 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add AWS Lamda testing for .NET 9 #6779

Add AWS Lamda testing for .NET 9 #6779

Uh oh!

andrewlock commented Mar 19, 2025

Uh oh!

datadog-datadog-prod-us1 bot commented Mar 19, 2025 •

edited

Loading

Uh oh!

andrewlock commented Mar 19, 2025

Uh oh!

andrewlock commented Mar 19, 2025

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6779

Raw results

Slower ⚠️ in #6779

Raw results

Uh oh!

Uh oh!

Uh oh!

Add AWS Lamda testing for .NET 9 #6779

Add AWS Lamda testing for .NET 9 #6779

Uh oh!

Conversation

andrewlock commented Mar 19, 2025

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

Uh oh!

datadog-datadog-prod-us1 bot commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Report

⌛ Performance Regressions vs Default Branch (6)

Uh oh!

andrewlock commented Mar 19, 2025

Execution-Time Benchmarks Report ⏱️

Uh oh!

andrewlock commented Mar 19, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6779

Raw results

Slower ⚠️ in #6779

Raw results

Uh oh!

Uh oh!

Uh oh!

datadog-datadog-prod-us1 bot commented Mar 19, 2025 •

edited

Loading