Skip to content

Disable code coverage unless forced #5090

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jan 22, 2024
Merged

Disable code coverage unless forced #5090

merged 1 commit into from
Jan 22, 2024

Conversation

andrewlock
Copy link
Member

Summary of changes

Disables coverlet code coverage everywhere unless explicitly requested.

Reason for change

The tests introduced in #4782 fail on master, because they verify the IL of some libraries, but coverlet rewrites the IL.

Implementation details

Stop running code coverage on master. We already skip it on branches because it contributed significantly to flake, and I don't think anyone checks the master results, so this will do for now.

We can do a fix for the IL comparison test to account for coverlet later, and then re-enable this if we wish.

Test coverage

We'll never know now 🙈

Other details

@andrewlock andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Jan 19, 2024
@andrewlock andrewlock requested a review from a team as a code owner January 19, 2024 11:59
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Jan 19, 2024 •
edited
Loading

Datadog Report

Branch report: andrew/ci/disable-coverlet
Commit report: 76f83b1
Test service: dd-trace-dotnet

✅ 0 Failed, 311220 Passed, 1506 Skipped, 57m 48.53s Wall Time

@andrewlock
Copy link
Member Author

andrewlock commented Jan 19, 2024 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5090) - mean (72ms)  : 62, 83
     .   : milestone, 72,
    master - mean (70ms)  : 62, 79
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (5090) - mean (957ms)  : 932, 983
     .   : milestone, 957,
    master - mean (955ms)  : 937, 973
     .   : milestone, 955,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5090) - mean (106ms)  : 104, 109
     .   : milestone, 106,
    master - mean (106ms)  : 103, 110
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (5090) - mean (696ms)  : 675, 717
     .   : milestone, 696,
    master - mean (698ms)  : 675, 721
     .   : milestone, 698,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5090) - mean (90ms)  : 86, 94
     .   : milestone, 90,
    master - mean (89ms)  : 85, 93
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (5090) - mean (651ms)  : 633, 670
     .   : milestone, 651,
    master - mean (653ms)  : 636, 670
     .   : milestone, 653,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5090) - mean (188ms)  : 186, 191
     .   : milestone, 188,
    master - mean (188ms)  : 185, 192
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (5090) - mean (1,061ms)  : 1037, 1086
     .   : milestone, 1061,
    master - mean (1,064ms)  : 1043, 1085
     .   : milestone, 1064,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5090) - mean (272ms)  : 268, 276
     .   : milestone, 272,
    master - mean (271ms)  : 266, 276
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (5090) - mean (1,052ms)  : 1025, 1080
     .   : milestone, 1052,
    master - mean (1,045ms)  : 1023, 1067
     .   : milestone, 1045,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5090) - mean (261ms)  : 258, 265
     .   : milestone, 261,
    master - mean (261ms)  : 257, 264
     .   : milestone, 261,

    section CallTarget+Inlining+NGEN
    This PR (5090) - mean (1,014ms)  : 990, 1038
     .   : milestone, 1014,
    master - mean (1,017ms)  : 997, 1037
     .   : milestone, 1017,
Loading

@andrewlock andrewlock force-pushed the andrew/ci/disable-coverlet branch from 79d7654 to 76f83b1 Compare January 19, 2024 12:45
@andrewlock
Copy link
Member Author

Benchmarks Report 🐌

Benchmarks for #5090 compared to master:

  • 3 benchmarks are faster, with geometric mean 1.133
  • 2 benchmarks have fewer allocations
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.88μs 49.3ns 304ns 0.0188 0.00938 0 7.49 KB
master StartStopWithChild netcoreapp3.1 10.9μs 58.7ns 316ns 0.0276 0.011 0 7.57 KB
master StartStopWithChild net472 17μs 73.4ns 284ns 1.34 0.341 0.119 7.95 KB
#5090 StartStopWithChild net6.0 8.81μs 48.6ns 279ns 0.0169 0.00845 0 7.48 KB
#5090 StartStopWithChild netcoreapp3.1 11μs 52.9ns 230ns 0.0266 0.0106 0 7.58 KB
#5090 StartStopWithChild net472 17.4μs 49.3ns 191ns 1.34 0.359 0.114 7.94 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 455μs 196ns 735ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 623μs 170ns 657ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 816μs 520ns 1.95μs 0.408 0 0 3.3 KB
#5090 WriteAndFlushEnrichedTraces net6.0 467μs 185ns 717ns 0 0 0 2.7 KB
#5090 WriteAndFlushEnrichedTraces netcoreapp3.1 621μs 282ns 1.06μs 0 0 0 2.7 KB
#5090 WriteAndFlushEnrichedTraces net472 796μs 280ns 1.05μs 0.396 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5090

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1 1.157 241.81 209.07

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 38.6μs 14.7ns 51ns 0.0193 0 0 1.77 KB
master AllCycleSimpleBody netcoreapp3.1 42.3μs 79.4ns 308ns 0.0208 0 0 1.74 KB
master AllCycleSimpleBody net472 44.7μs 71.4ns 258ns 0.288 0 0 1.81 KB
master AllCycleMoreComplexBody net6.0 198μs 110ns 427ns 0.0989 0 0 9.25 KB
master AllCycleMoreComplexBody netcoreapp3.1 211μs 125ns 483ns 0.106 0 0 9.14 KB
master AllCycleMoreComplexBody net472 224μs 77ns 288ns 1.45 0 0 9.32 KB
master ObjectExtractorSimpleBody net6.0 145ns 0.0748ns 0.259ns 0.00397 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 242ns 0.132ns 0.513ns 0.00378 0 0 272 B
master ObjectExtractorSimpleBody net472 166ns 0.163ns 0.609ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 2.94μs 2.4ns 8.99ns 0.0529 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 3.98μs 1.28ns 4.61ns 0.0498 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 3.81μs 4.32ns 16.2ns 0.602 0.00571 0 3.8 KB
#5090 AllCycleSimpleBody net6.0 39.4μs 63.5ns 246ns 0.0196 0 0 1.77 KB
#5090 AllCycleSimpleBody netcoreapp3.1 42.2μs 83.6ns 324ns 0.0211 0 0 1.74 KB
#5090 AllCycleSimpleBody net472 44.3μs 15.6ns 60.5ns 0.288 0 0 1.81 KB
#5090 AllCycleMoreComplexBody net6.0 202μs 98.2ns 380ns 0.1 0 0 9.25 KB
#5090 AllCycleMoreComplexBody netcoreapp3.1 214μs 310ns 1.2μs 0.106 0 0 9.14 KB
#5090 AllCycleMoreComplexBody net472 227μs 132ns 493ns 1.47 0 0 9.32 KB
#5090 ObjectExtractorSimpleBody net6.0 139ns 0.155ns 0.579ns 0.00393 0 0 280 B
#5090 ObjectExtractorSimpleBody netcoreapp3.1 209ns 0.186ns 0.697ns 0.00377 0 0 272 B
#5090 ObjectExtractorSimpleBody net472 184ns 0.198ns 0.766ns 0.0445 0 0 281 B
#5090 ObjectExtractorMoreComplexBody net6.0 2.98μs 2.22ns 8.61ns 0.0524 0 0 3.78 KB
#5090 ObjectExtractorMoreComplexBody netcoreapp3.1 4.1μs 1.88ns 7.29ns 0.0492 0 0 3.69 KB
#5090 ObjectExtractorMoreComplexBody net472 3.76μs 2.17ns 8.13ns 0.602 0.00563 0 3.8 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5090

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (100))‑net6.0 1.125 110,036.88 97,800.82

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWaf(args=NestedMap (10)) net6.0 50.8μs 21ns 78.4ns 0.226 0 0 16.06 KB
master RunWaf(args=NestedMap (10)) netcoreapp3.1 70.1μs 179ns 692ns 0.206 0 0 16.06 KB
master RunWaf(args=NestedMap (10)) net472 98.2μs 495ns 2.16μs 2.55 0.1 0 16.14 KB
master RunWafTwice(args=NestedMap (10)) net6.0 52.3μs 22.4ns 83.9ns 0.236 0 0 16.6 KB
master RunWafTwice(args=NestedMap (10)) netcoreapp3.1 75μs 385ns 1.72μs 0.19 0 0 16.58 KB
master RunWafTwice(args=NestedMap (10)) net472 104μs 49.8ns 193ns 2.62 0.105 0 16.69 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 119μs 57.4ns 222ns 0.326 0 0 22.41 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 129μs 673ns 3.3μs 0.255 0 0 22.36 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 163μs 112ns 432ns 3.54 0.157 0 22.7 KB
master RunWaf(args=NestedMap (100)) net6.0 110μs 53.3ns 200ns 0.43 0 0 32.76 KB
master RunWaf(args=NestedMap (100)) netcoreapp3.1 135μs 66.9ns 232ns 0.443 0 0 33.33 KB
master RunWaf(args=NestedMap (100)) net472 189μs 709ns 2.75μs 5.33 0.374 0 33.67 KB
master RunWafTwice(args=NestedMap (100)) net6.0 103μs 555ns 3.09μs 0.449 0 0 33.3 KB
master RunWafTwice(args=NestedMap (100)) netcoreapp3.1 142μs 478ns 1.85μs 0.424 0 0 33.86 KB
master RunWafTwice(args=NestedMap (100)) net472 201μs 61.3ns 237ns 5.35 0.389 0 34.23 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 172μs 105ns 405ns 0.551 0 0 39.1 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 198μs 761ns 2.95μs 0.486 0 0 39.63 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net472 253μs 159ns 614ns 6.36 0.509 0 40.23 KB
master RunWaf(args=NestedMap (20)) net6.0 99.6μs 43.3ns 189ns 0.43 0 0 32.18 KB
master RunWaf(args=NestedMap (20)) netcoreapp3.1 134μs 753ns 4.82μs 0.441 0 0 32.3 KB
master RunWaf(args=NestedMap (20)) net472 183μs 72.3ns 271ns 5.1 0.364 0 32.63 KB
master RunWafTwice(args=NestedMap (20)) net6.0 103μs 453ns 1.75μs 0.469 0 0 32.72 KB
master RunWafTwice(args=NestedMap (20)) netcoreapp3.1 139μs 749ns 4.3μs 0.427 0 0 32.82 KB
master RunWafTwice(args=NestedMap (20)) net472 197μs 729ns 2.82μs 5.21 0.386 0 33.19 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 166μs 94.7ns 367ns 0.557 0 0 38.53 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 197μs 90ns 349ns 0.491 0 0 38.6 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 251μs 114ns 428ns 6.16 0.503 0 39.2 KB
#5090 RunWaf(args=NestedMap (10)) net6.0 55.9μs 12.8ns 47.8ns 0.202 0 0 16.06 KB
#5090 RunWaf(args=NestedMap (10)) netcoreapp3.1 68.8μs 98.1ns 380ns 0.202 0 0 16.06 KB
#5090 RunWaf(args=NestedMap (10)) net472 97.7μs 41.2ns 159ns 2.52 0.095 0 16.14 KB
#5090 RunWafTwice(args=NestedMap (10)) net6.0 56.2μs 21.4ns 80.1ns 0.226 0 0 16.6 KB
#5090 RunWafTwice(args=NestedMap (10)) netcoreapp3.1 74.1μs 388ns 1.9μs 0.225 0 0 16.58 KB
#5090 RunWafTwice(args=NestedMap (10)) net472 109μs 52.8ns 205ns 2.65 0.104 0 16.69 KB
#5090 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 109μs 37.4ns 140ns 0.27 0 0 22.41 KB
#5090 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 126μs 354ns 1.23μs 0.249 0 0 22.36 KB
#5090 RunWafWithAttack(args=Neste(...)tack) [22]) net472 164μs 191ns 739ns 3.57 0.159 0 22.7 KB
#5090 RunWaf(args=NestedMap (100)) net6.0 97.8μs 29.3ns 109ns 0.443 0 0 32.76 KB
#5090 RunWaf(args=NestedMap (100)) netcoreapp3.1 131μs 512ns 1.98μs 0.458 0 0 33.33 KB
#5090 RunWaf(args=NestedMap (100)) net472 185μs 211ns 791ns 5.35 0.369 0 33.67 KB
#5090 RunWafTwice(args=NestedMap (100)) net6.0 112μs 580ns 2.84μs 0.482 0 0 33.3 KB
#5090 RunWafTwice(args=NestedMap (100)) netcoreapp3.1 137μs 521ns 2.02μs 0.464 0 0 33.86 KB
#5090 RunWafTwice(args=NestedMap (100)) net472 193μs 89.5ns 347ns 5.4 0.386 0 34.23 KB
#5090 RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 158μs 56.5ns 226ns 0.507 0 0 39.1 KB
#5090 RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 207μs 405ns 1.57μs 0.488 0 0 39.63 KB
#5090 RunWafWithAttack(args=Neste(...)tack) [23]) net472 252μs 131ns 508ns 6.29 0.503 0 40.23 KB
#5090 RunWaf(args=NestedMap (20)) net6.0 102μs 48.5ns 188ns 0.41 0 0 32.18 KB
#5090 RunWaf(args=NestedMap (20)) netcoreapp3.1 132μs 548ns 1.98μs 0.381 0 0 32.3 KB
#5090 RunWaf(args=NestedMap (20)) net472 182μs 85.1ns 329ns 5.19 0.364 0 32.63 KB
#5090 RunWafTwice(args=NestedMap (20)) net6.0 106μs 60ns 232ns 0.429 0 0 32.72 KB
#5090 RunWafTwice(args=NestedMap (20)) netcoreapp3.1 137μs 414ns 1.49μs 0.413 0 0 32.82 KB
#5090 RunWafTwice(args=NestedMap (20)) net472 193μs 111ns 431ns 5.18 0.384 0 33.19 KB
#5090 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 173μs 164ns 633ns 0.483 0 0 38.53 KB
#5090 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 197μs 747ns 2.89μs 0.506 0 0 38.6 KB
#5090 RunWafWithAttack(args=Neste(...)tack) [22]) net472 250μs 110ns 427ns 6.12 0.5 0 39.2 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 175μs 153ns 594ns 0.174 0 0 18.25 KB
master SendRequest netcoreapp3.1 195μs 302ns 1.17μs 0.196 0 0 20.41 KB
master SendRequest net472 0.000187ns 0.000166ns 0.000621ns 0 0 0 0 b
#5090 SendRequest net6.0 172μs 209ns 808ns 0.256 0 0 18.25 KB
#5090 SendRequest netcoreapp3.1 191μs 256ns 992ns 0.191 0 0 20.41 KB
#5090 SendRequest net472 0.00071ns 0.000284ns 0.0011ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 543μs 2.33μs 9.04μs 0.558 0 0 41.45 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 641μs 516ns 2μs 0.321 0 0 41.85 KB
master WriteAndFlushEnrichedTraces net472 865μs 4.46μs 20.9μs 8.45 2.53 0.422 53.24 KB
#5090 WriteAndFlushEnrichedTraces net6.0 548μs 1.01μs 3.92μs 0.548 0 0 41.5 KB
#5090 WriteAndFlushEnrichedTraces netcoreapp3.1 653μs 325ns 1.17μs 0.332 0 0 41.88 KB
#5090 WriteAndFlushEnrichedTraces net472 832μs 3.53μs 13.7μs 8.25 2.36 0.393 53.22 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.22μs 0.642ns 2.49ns 0.0109 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.46μs 0.797ns 3.09ns 0.0104 0 0 768 B
master ExecuteNonQuery net472 1.81μs 1.42ns 5.1ns 0.116 0 0 730 B
#5090 ExecuteNonQuery net6.0 1.16μs 0.506ns 1.89ns 0.0104 0 0 768 B
#5090 ExecuteNonQuery netcoreapp3.1 1.51μs 0.604ns 2.26ns 0.0106 0 0 768 B
#5090 ExecuteNonQuery net472 1.76μs 2.21ns 7.98ns 0.116 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.21μs 0.96ns 3.72ns 0.0133 0 0 936 B
master CallElasticsearch netcoreapp3.1 1.5μs 0.727ns 2.82ns 0.0121 0 0 936 B
master CallElasticsearch net472 2.61μs 1.72ns 6.68ns 0.151 0 0 955 B
master CallElasticsearchAsync net6.0 1.28μs 0.439ns 1.64ns 0.0125 0 0 912 B
master CallElasticsearchAsync netcoreapp3.1 1.56μs 0.658ns 2.46ns 0.0133 0 0 984 B
master CallElasticsearchAsync net472 2.6μs 1.04ns 4.05ns 0.16 0 0 1.01 KB
#5090 CallElasticsearch net6.0 1.21μs 1.08ns 4.03ns 0.0127 0 0 936 B
#5090 CallElasticsearch netcoreapp3.1 1.62μs 2.01ns 7.53ns 0.0129 0 0 936 B
#5090 CallElasticsearch net472 2.42μs 0.786ns 2.94ns 0.151 0 0 955 B
#5090 CallElasticsearchAsync net6.0 1.19μs 0.434ns 1.68ns 0.013 0 0 912 B
#5090 CallElasticsearchAsync netcoreapp3.1 1.65μs 1.39ns 5.21ns 0.0132 0 0 984 B
#5090 CallElasticsearchAsync net472 2.72μs 0.51ns 1.84ns 0.16 0 0 1.01 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.29μs 0.89ns 3.45ns 0.0131 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.67μs 1.56ns 6.03ns 0.0126 0 0 912 B
master ExecuteAsync net472 1.78μs 1.12ns 4.19ns 0.139 0 0 875 B
#5090 ExecuteAsync net6.0 1.24μs 0.548ns 2.05ns 0.0131 0 0 912 B
#5090 ExecuteAsync netcoreapp3.1 1.66μs 1.51ns 5.63ns 0.0124 0 0 912 B
#5090 ExecuteAsync net472 1.81μs 0.93ns 3.48ns 0.138 0 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 3.93μs 3.21ns 12.4ns 0.0293 0 0 2.1 KB
master SendAsync netcoreapp3.1 4.94μs 23.2ns 125ns 0.0341 0 0 2.63 KB
master SendAsync net472 7.8μs 8.05ns 31.2ns 0.524 0 0 3.31 KB
#5090 SendAsync net6.0 4.11μs 2.45ns 9.47ns 0.0285 0 0 2.1 KB
#5090 SendAsync netcoreapp3.1 4.91μs 1.16ns 4.5ns 0.0346 0 0 2.63 KB
#5090 SendAsync net472 7.73μs 3.25ns 11.3ns 0.524 0 0 3.31 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #5090

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0 202.62 KB 204.83 KB 2.22 KB 1.09%

Fewer allocations 🎉 in #5090

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 212.78 KB 211.55 KB -1.23 KB -0.58%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 57.67 KB 55.83 KB -1.84 KB -3.19%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 58.1μs 602ns 5.93μs 0 0 0 43.44 KB
master StringConcatBenchmark netcoreapp3.1 53.9μs 260ns 1.01μs 0 0 0 42.64 KB
master StringConcatBenchmark net472 38.1μs 91.8ns 344ns 0 0 0 57.67 KB
master StringConcatAspectBenchmark net6.0 272μs 4.72μs 43.6μs 0 0 0 202.62 KB
master StringConcatAspectBenchmark netcoreapp3.1 293μs 5.49μs 50.6μs 0 0 0 212.78 KB
master StringConcatAspectBenchmark net472 245μs 4.46μs 42.3μs 0 0 0 221.18 KB
#5090 StringConcatBenchmark net6.0 53.6μs 249ns 964ns 0 0 0 43.44 KB
#5090 StringConcatBenchmark netcoreapp3.1 53.7μs 255ns 956ns 0 0 0 42.64 KB
#5090 StringConcatBenchmark net472 38μs 94.1ns 339ns 0 0 0 55.83 KB
#5090 StringConcatAspectBenchmark net6.0 287μs 6.22μs 59μs 0 0 0 204.83 KB
#5090 StringConcatAspectBenchmark netcoreapp3.1 307μs 3.11μs 28μs 0 0 0 211.55 KB
#5090 StringConcatAspectBenchmark net472 263μs 6.2μs 60.7μs 0 0 0 221.18 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.44μs 0.961ns 3.47ns 0.0223 0 0 1.57 KB
master EnrichedLog netcoreapp3.1 2.28μs 1.73ns 6.47ns 0.0205 0 0 1.57 KB
master EnrichedLog net472 2.72μs 3.3ns 12.8ns 0.237 0 0 1.5 KB
#5090 EnrichedLog net6.0 1.43μs 0.656ns 2.54ns 0.0216 0 0 1.57 KB
#5090 EnrichedLog netcoreapp3.1 2.14μs 0.892ns 3.46ns 0.0204 0 0 1.57 KB
#5090 EnrichedLog net472 2.69μs 2.52ns 9.41ns 0.238 0 0 1.5 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 168ns 652ns 0 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 118μs 113ns 438ns 0 0 0 4.21 KB
master EnrichedLog net472 149μs 42.4ns 153ns 0.669 0.223 0 4.39 KB
#5090 EnrichedLog net6.0 115μs 168ns 649ns 0.0573 0 0 4.21 KB
#5090 EnrichedLog netcoreapp3.1 118μs 97.5ns 351ns 0 0 0 4.21 KB
#5090 EnrichedLog net472 148μs 88.5ns 343ns 0.672 0.224 0 4.39 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.07μs 2.17ns 8.42ns 0.0292 0 0 2.13 KB
master EnrichedLog netcoreapp3.1 4.3μs 2.37ns 8.22ns 0.0279 0 0 2.13 KB
master EnrichedLog net472 4.76μs 1.27ns 4.57ns 0.308 0 0 1.95 KB
#5090 EnrichedLog net6.0 3.17μs 0.703ns 2.72ns 0.0299 0 0 2.13 KB
#5090 EnrichedLog netcoreapp3.1 4.19μs 2.16ns 7.79ns 0.0294 0 0 2.13 KB
#5090 EnrichedLog net472 4.83μs 1.75ns 6.79ns 0.309 0 0 1.95 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.35μs 1.69ns 6.55ns 0.0155 0 0 1.1 KB
master SendReceive netcoreapp3.1 1.74μs 1.3ns 4.88ns 0.0148 0 0 1.1 KB
master SendReceive net472 2.18μs 2.41ns 9.34ns 0.177 0 0 1.12 KB
#5090 SendReceive net6.0 1.29μs 0.633ns 2.45ns 0.0155 0 0 1.1 KB
#5090 SendReceive netcoreapp3.1 1.76μs 0.948ns 3.55ns 0.0142 0 0 1.1 KB
#5090 SendReceive net472 2.12μs 0.913ns 3.41ns 0.176 0 0 1.12 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.71μs 0.959ns 3.71ns 0.0216 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.97μs 1.11ns 4.17ns 0.0198 0 0 1.58 KB
master EnrichedLog net472 4.32μs 1.13ns 4.22ns 0.311 0 0 1.97 KB
#5090 EnrichedLog net6.0 2.66μs 1.39ns 5.2ns 0.0212 0 0 1.53 KB
#5090 EnrichedLog netcoreapp3.1 4.03μs 1.55ns 6.02ns 0.0199 0 0 1.58 KB
#5090 EnrichedLog net472 4.31μs 2.18ns 8.44ns 0.311 0 0 1.97 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 513ns 0.127ns 0.476ns 0.00738 0 0 536 B
master StartFinishSpan netcoreapp3.1 739ns 0.887ns 3.32ns 0.00731 0 0 536 B
master StartFinishSpan net472 788ns 0.314ns 1.17ns 0.0853 0 0 538 B
master StartFinishScope net6.0 608ns 0.267ns 1.03ns 0.00914 0 0 656 B
master StartFinishScope netcoreapp3.1 811ns 0.31ns 1.2ns 0.00848 0 0 656 B
master StartFinishScope net472 936ns 1.05ns 4.09ns 0.0981 0 0 618 B
#5090 StartFinishSpan net6.0 461ns 0.149ns 0.577ns 0.00743 0 0 536 B
#5090 StartFinishSpan netcoreapp3.1 721ns 0.683ns 2.46ns 0.00713 0 0 536 B
#5090 StartFinishSpan net472 744ns 0.289ns 1.12ns 0.0852 0 0 538 B
#5090 StartFinishScope net6.0 660ns 0.337ns 1.31ns 0.00927 0 0 656 B
#5090 StartFinishScope netcoreapp3.1 805ns 0.296ns 1.15ns 0.00861 0 0 656 B
#5090 StartFinishScope net472 898ns 0.584ns 2.26ns 0.0979 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5090

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net472 1.119 1,232.74 1,101.77

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 748ns 0.473ns 1.83ns 0.00935 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 863ns 2.12ns 8.2ns 0.00884 0 0 656 B
master RunOnMethodBegin net472 1.23μs 0.347ns 1.34ns 0.0978 0 0 618 B
#5090 RunOnMethodBegin net6.0 763ns 0.23ns 0.892ns 0.00919 0 0 656 B
#5090 RunOnMethodBegin netcoreapp3.1 928ns 0.524ns 2.03ns 0.00876 0 0 656 B
#5090 RunOnMethodBegin net472 1.1μs 0.716ns 2.77ns 0.0979 0 0 618 B

@andrewlock
Copy link
Member Author

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5090) (11.807M)   : 0, 11807358
    master (11.998M)   : 0, 11998376
    benchmarks/2.9.0 (11.682M)   : 0, 11681575

    section Automatic
    This PR (5090) (8.031M)   : 0, 8031383
    master (8.129M)   : 0, 8129379
    benchmarks/2.9.0 (8.446M)   : 0, 8445739

    section Trace stats
    This PR (5090) (8.416M)   : 0, 8416472
    master (8.529M)   : 0, 8529273

    section Manual
    This PR (5090) (10.345M)   : 0, 10345080
    master (10.271M)   : 0, 10271133

    section Manual + Automatic
    This PR (5090) (7.540M)   : 0, 7539819
    master (7.591M)   : 0, 7590530

    section Version Conflict
    This PR (5090) (6.859M)   : 0, 6859016
    master (6.907M)   : 0, 6906785
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5090) (9.517M)   : 0, 9516527
    master (9.594M)   : 0, 9593658
    benchmarks/2.9.0 (9.657M)   : 0, 9656830

    section Automatic
    This PR (5090) (6.591M)   : 0, 6591314
    master (6.574M)   : 0, 6573613

    section Trace stats
    This PR (5090) (6.768M)   : 0, 6767639
    master (6.962M)   : 0, 6961649

    section Manual
    This PR (5090) (8.292M)   : 0, 8291581
    master (8.281M)   : 0, 8281219

    section Manual + Automatic
    This PR (5090) (6.148M)   : 0, 6147872
    master (6.258M)   : 0, 6258314

    section Version Conflict
    This PR (5090) (5.630M)   : 0, 5629871
    master (5.552M)   : 0, 5551784
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5090) (10.108M)   : 0, 10107552
    master (10.243M)   : 0, 10242543
    benchmarks/2.9.0 (10.207M)   : 0, 10206890

    section Automatic
    This PR (5090) (7.023M)   : 0, 7022735
    master (7.076M)   : 0, 7076485
    benchmarks/2.9.0 (7.362M)   : 0, 7361712

    section Trace stats
    This PR (5090) (7.352M)   : 0, 7351588
    master (7.453M)   : 0, 7453286

    section Manual
    This PR (5090) (8.718M)   : 0, 8717819
    master (8.806M)   : 0, 8806398

    section Manual + Automatic
    This PR (5090) (6.805M)   : 0, 6804692
    master (6.909M)   : 0, 6909369

    section Version Conflict
    This PR (5090) (6.106M)   : 0, 6106225
    master (6.090M)   : 0, 6090260
Loading 
gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.762M)   : 0, 7761523
    benchmarks/2.9.0 (7.448M)   : 0, 7447634

    section No attack
    master (1.792M)   : 0, 1791629
    benchmarks/2.9.0 (3.135M)   : 0, 3134564

    section Attack
    master (1.416M)   : 0, 1415708
    benchmarks/2.9.0 (2.468M)   : 0, 2467773

    section Blocking
    master (3.120M)   : 0, 3120029

    section IAST default
    master (6.490M)   : 0, 6489771

    section IAST full
    master (5.606M)   : 0, 5605809

    section Base vuln
    master (0.943M)   : 0, 943169

    section IAST vuln
    master (0.885M)   : 0, 885243
Loading

@andrewlock andrewlock merged commit e4903c7 into master Jan 22, 2024
@andrewlock andrewlock deleted the andrew/ci/disable-coverlet branch January 22, 2024 09:52
@github-actions github-actions bot added this to the vNext milestone Jan 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
1 more reviewer

@kevingosse kevingosse kevingosse approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
area:builds project files, build scripts, pipelines, versioning, releases, packages
Projects
None yet
Milestone
2.46.0
Development

Successfully merging this pull request may close these issues.
2 participants
@andrewlock @kevingosse