Skip to content

Minor CI fixes #5056

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Jan 18, 2024
Merged

Minor CI fixes #5056

merged 3 commits into from
Jan 18, 2024

Conversation

andrewlock
Copy link
Member

Summary of changes

  • Remove duplicate collection definition
  • Fix using unserializable data in tests

Reason for change

xunit was warning about them. The latter one means we weren't actually running all those tests

Implementation details

Use SerializableDictionary and remove the duplicate

Test coverage

Should be covered!

Other details

@andrewlock andrewlock added the area:tests unit tests, integration tests label Jan 12, 2024
@andrewlock andrewlock requested review from a team as code owners January 12, 2024 17:06
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Jan 12, 2024 •
edited
Loading

Datadog Report

Branch report: andrew/test-fixes
Commit report: d272633
Test service: dd-trace-dotnet

✅ 0 Failed, 304007 Passed, 1015 Skipped, 49m 11.76s Wall Time

@andrewlock
Copy link
Member Author

andrewlock commented Jan 12, 2024 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5056) - mean (72ms)  : 61, 82
     .   : milestone, 72,
    master - mean (70ms)  : 61, 78
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (5056) - mean (1,029ms)  : 1007, 1051
     .   : milestone, 1029,
    master - mean (1,019ms)  : 995, 1042
     .   : milestone, 1019,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5056) - mean (107ms)  : 103, 110
     .   : milestone, 107,
    master - mean (105ms)  : 100, 110
     .   : milestone, 105,

    section CallTarget+Inlining+NGEN
    This PR (5056) - mean (741ms)  : 716, 766
     .   : milestone, 741,
    master - mean (729ms)  : 708, 751
     .   : milestone, 729,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5056) - mean (89ms)  : 84, 93
     .   : milestone, 89,
    master - mean (89ms)  : 84, 93
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (5056) - mean (689ms)  : 666, 713
     .   : milestone, 689,
    master - mean (684ms)  : 663, 706
     .   : milestone, 684,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5056) - mean (188ms)  : 185, 190
     .   : milestone, 188,
    master - mean (188ms)  : 185, 191
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (5056) - mean (1,132ms)  : 1108, 1156
     .   : milestone, 1132,
    master - mean (1,133ms)  : 1105, 1160
     .   : milestone, 1133,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5056) - mean (271ms)  : 267, 275
     .   : milestone, 271,
    master - mean (272ms)  : 268, 276
     .   : milestone, 272,

    section CallTarget+Inlining+NGEN
    This PR (5056) - mean (1,085ms)  : 1064, 1106
     .   : milestone, 1085,
    master - mean (1,093ms)  : 1072, 1114
     .   : milestone, 1093,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5056) - mean (261ms)  : 258, 263
     .   : milestone, 261,
    master - mean (262ms)  : 257, 267
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (5056) - mean (1,053ms)  : 1031, 1076
     .   : milestone, 1053,
    master - mean (1,055ms)  : 1028, 1083
     .   : milestone, 1055,
Loading

@andrewlock
Copy link
Member Author

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5056) (11.092M)   : 0, 11092249
    master (11.893M)   : 0, 11892622
    benchmarks/2.9.0 (11.642M)   : 0, 11641842

    section Automatic
    This PR (5056) (7.535M)   : crit ,0, 7534513
    master (8.099M)   : 0, 8098638
    benchmarks/2.9.0 (8.382M)   : 0, 8382125

    section Trace stats
    This PR (5056) (7.823M)   : 0, 7823391
    master (5.895M)   : 0, 5895492

    section Manual
    This PR (5056) (9.530M)   : crit ,0, 9530457
    master (10.361M)   : 0, 10361124

    section Manual + Automatic
    This PR (5056) (7.065M)   : crit ,0, 7064785
    master (7.670M)   : 0, 7670088

    section Version Conflict
    This PR (5056) (6.349M)   : crit ,0, 6348724
    master (6.859M)   : 0, 6859229
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5056) (9.565M)   : 0, 9564575
    master (9.617M)   : 0, 9616856
    benchmarks/2.9.0 (9.719M)   : 0, 9718800

    section Automatic
    This PR (5056) (6.487M)   : 0, 6486953
    master (6.615M)   : 0, 6615394

    section Trace stats
    This PR (5056) (6.869M)   : 0, 6868596
    master (6.934M)   : 0, 6934299

    section Manual
    This PR (5056) (8.242M)   : 0, 8242160
    master (8.540M)   : 0, 8539968

    section Manual + Automatic
    This PR (5056) (6.146M)   : 0, 6145951
    master (6.148M)   : 0, 6147793

    section Version Conflict
    This PR (5056) (5.699M)   : 0, 5698833
    master (5.693M)   : 0, 5693193
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5056) (10.012M)   : 0, 10012382
    benchmarks/2.9.0 (9.943M)   : 0, 9943294

    section Automatic
    This PR (5056) (7.044M)   : 0, 7044417
    benchmarks/2.9.0 (7.260M)   : 0, 7260428

    section Trace stats
    This PR (5056) (7.338M)   : 0, 7337604

    section Manual
    This PR (5056) (8.643M)   : 0, 8642652

    section Manual + Automatic
    This PR (5056) (6.786M)   : 0, 6786096

    section Version Conflict
    This PR (5056) (6.152M)   : 0, 6152242
Loading 
gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.496M)   : 0, 7496114
    benchmarks/2.9.0 (7.836M)   : 0, 7835831

    section No attack
    master (1.764M)   : 0, 1763502
    benchmarks/2.9.0 (3.226M)   : 0, 3226182

    section Attack
    master (1.399M)   : 0, 1399328
    benchmarks/2.9.0 (2.474M)   : 0, 2473872

    section Blocking
    master (3.098M)   : 0, 3097902

    section IAST default
    master (6.459M)   : 0, 6458516

    section IAST full
    master (5.769M)   : 0, 5769139

    section Base vuln
    master (0.971M)   : 0, 970670

    section IAST vuln
    master (0.901M)   : 0, 900962
Loading

@andrewlock
Copy link
Member Author

andrewlock commented Jan 13, 2024 •
edited
Loading

Benchmarks Report 🐌

Benchmarks for #5056 compared to master:

  • 6 benchmarks are faster, with geometric mean 1.163
  • 2 benchmarks are slower, with geometric mean 1.135
  • 2 benchmarks have fewer allocations
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.68μs 44.8ns 224ns 0.0182 0.00909 0 7.48 KB
master StartStopWithChild netcoreapp3.1 10.8μs 53.5ns 307ns 0.0266 0.0106 0 7.57 KB
master StartStopWithChild net472 17.1μs 52.1ns 202ns 1.34 0.341 0.111 7.95 KB
#5056 StartStopWithChild net6.0 8.49μs 46.6ns 276ns 0.0292 0.0167 0.00418 7.48 KB
#5056 StartStopWithChild netcoreapp3.1 10.9μs 59.6ns 362ns 0.021 0.0105 0 7.58 KB
#5056 StartStopWithChild net472 17.1μs 64.1ns 231ns 1.37 0.372 0.104 7.95 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 479μs 331ns 1.28μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 628μs 196ns 732ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 791μs 361ns 1.4μs 0.396 0 0 3.3 KB
#5056 WriteAndFlushEnrichedTraces net6.0 454μs 360ns 1.39μs 0 0 0 2.7 KB
#5056 WriteAndFlushEnrichedTraces netcoreapp3.1 631μs 159ns 594ns 0 0 0 2.7 KB
#5056 WriteAndFlushEnrichedTraces net472 793μs 290ns 1.12μs 0.396 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 37.8μs 28.7ns 111ns 0.0191 0 0 1.77 KB
master AllCycleSimpleBody netcoreapp3.1 42.2μs 30.2ns 117ns 0.021 0 0 1.74 KB
master AllCycleSimpleBody net472 44.3μs 15.5ns 60.2ns 0.267 0 0 1.81 KB
master AllCycleMoreComplexBody net6.0 200μs 60.2ns 217ns 0.0994 0 0 9.25 KB
master AllCycleMoreComplexBody netcoreapp3.1 212μs 183ns 707ns 0.106 0 0 9.14 KB
master AllCycleMoreComplexBody net472 225μs 99.6ns 386ns 1.39 0 0 9.32 KB
master ObjectExtractorSimpleBody net6.0 137ns 0.0319ns 0.119ns 0.00397 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 194ns 0.112ns 0.419ns 0.00371 0 0 272 B
master ObjectExtractorSimpleBody net472 173ns 0.141ns 0.544ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 2.95μs 1.02ns 3.96ns 0.053 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 4.24μs 1.8ns 6.5ns 0.0488 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 3.78μs 2.07ns 8.01ns 0.603 0.00567 0 3.8 KB
#5056 AllCycleSimpleBody net6.0 38.7μs 17.7ns 68.6ns 0.0193 0 0 1.77 KB
#5056 AllCycleSimpleBody netcoreapp3.1 41.5μs 40ns 155ns 0.0207 0 0 1.74 KB
#5056 AllCycleSimpleBody net472 45μs 32.2ns 125ns 0.271 0 0 1.81 KB
#5056 AllCycleMoreComplexBody net6.0 200μs 75.3ns 261ns 0.0998 0 0 9.25 KB
#5056 AllCycleMoreComplexBody netcoreapp3.1 211μs 262ns 1.01μs 0.105 0 0 9.14 KB
#5056 AllCycleMoreComplexBody net472 227μs 61.4ns 230ns 1.47 0 0 9.32 KB
#5056 ObjectExtractorSimpleBody net6.0 138ns 0.0556ns 0.208ns 0.00393 0 0 280 B
#5056 ObjectExtractorSimpleBody netcoreapp3.1 200ns 0.0766ns 0.297ns 0.00372 0 0 272 B
#5056 ObjectExtractorSimpleBody net472 167ns 0.0975ns 0.365ns 0.0446 0 0 281 B
#5056 ObjectExtractorMoreComplexBody net6.0 3.03μs 1.08ns 4.2ns 0.0529 0 0 3.78 KB
#5056 ObjectExtractorMoreComplexBody netcoreapp3.1 4.09μs 1.67ns 6.47ns 0.049 0 0 3.69 KB
#5056 ObjectExtractorMoreComplexBody net472 3.81μs 7.35ns 28.5ns 0.602 0.00571 0 3.8 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5056

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (10))‑net6.0 1.123 57,773.17 51,428.09

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWaf(args=NestedMap (10)) net6.0 57.8μs 33.2ns 129ns 0.23 0 0 16.06 KB
master RunWaf(args=NestedMap (10)) netcoreapp3.1 66.1μs 362ns 2.17μs 0.21 0 0 16.06 KB
master RunWaf(args=NestedMap (10)) net472 101μs 60.2ns 233ns 2.55 0.0964 0 16.14 KB
master RunWafTwice(args=NestedMap (10)) net6.0 56.7μs 28.7ns 111ns 0.209 0 0 16.6 KB
master RunWafTwice(args=NestedMap (10)) netcoreapp3.1 73.1μs 374ns 1.87μs 0.208 0 0 16.58 KB
master RunWafTwice(args=NestedMap (10)) net472 108μs 505ns 1.95μs 2.64 0.104 0 16.69 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 115μs 405ns 1.57μs 0.288 0 0 22.41 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 129μs 672ns 3.08μs 0.25 0 0 22.36 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 165μs 302ns 1.17μs 3.54 0.157 0 22.7 KB
master RunWaf(args=NestedMap (100)) net6.0 100μs 33.2ns 120ns 0.431 0 0 32.76 KB
master RunWaf(args=NestedMap (100)) netcoreapp3.1 135μs 760ns 5.15μs 0.414 0 0 33.33 KB
master RunWaf(args=NestedMap (100)) net472 194μs 67.8ns 263ns 5.32 0.373 0 33.67 KB
master RunWafTwice(args=NestedMap (100)) net6.0 106μs 601ns 4.2μs 0.452 0 0 33.3 KB
master RunWafTwice(args=NestedMap (100)) netcoreapp3.1 135μs 668ns 2.75μs 0.461 0 0 33.86 KB
master RunWafTwice(args=NestedMap (100)) net472 196μs 259ns 1μs 5.35 0.389 0 34.22 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 159μs 61.9ns 240ns 0.557 0 0 39.1 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 200μs 1.01μs 4.38μs 0.491 0 0 39.63 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net472 252μs 143ns 554ns 6.31 0.505 0 40.23 KB
master RunWaf(args=NestedMap (20)) net6.0 104μs 555ns 3.04μs 0.42 0 0 32.18 KB
master RunWaf(args=NestedMap (20)) netcoreapp3.1 132μs 717ns 3.86μs 0.396 0 0 32.3 KB
master RunWaf(args=NestedMap (20)) net472 186μs 867ns 3.47μs 5.12 0.366 0 32.63 KB
master RunWafTwice(args=NestedMap (20)) net6.0 108μs 215ns 831ns 0.436 0 0 32.72 KB
master RunWafTwice(args=NestedMap (20)) netcoreapp3.1 141μs 701ns 2.89μs 0.414 0 0 32.82 KB
master RunWafTwice(args=NestedMap (20)) net472 195μs 101ns 392ns 5.27 0.39 0 33.19 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 158μs 93.9ns 325ns 0.55 0 0 38.53 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 195μs 632ns 2.45μs 0.479 0 0 38.6 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 258μs 934ns 3.62μs 6.17 0.504 0 39.2 KB
#5056 RunWaf(args=NestedMap (10)) net6.0 51.4μs 22.4ns 77.6ns 0.206 0 0 16.06 KB
#5056 RunWaf(args=NestedMap (10)) netcoreapp3.1 67.9μs 373ns 2.18μs 0.182 0 0 16.06 KB
#5056 RunWaf(args=NestedMap (10)) net472 98.4μs 512ns 2.45μs 2.52 0.0968 0 16.14 KB
#5056 RunWafTwice(args=NestedMap (10)) net6.0 57.5μs 330ns 2.93μs 0.236 0 0 16.6 KB
#5056 RunWafTwice(args=NestedMap (10)) netcoreapp3.1 70.1μs 100ns 346ns 0.209 0 0 16.58 KB
#5056 RunWafTwice(args=NestedMap (10)) net472 107μs 521ns 2.02μs 2.62 0.103 0 16.69 KB
#5056 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 111μs 503ns 1.95μs 0.272 0 0 22.41 KB
#5056 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 130μs 398ns 1.54μs 0.253 0 0 22.36 KB
#5056 RunWafWithAttack(args=Neste(...)tack) [22]) net472 159μs 229ns 885ns 3.56 0.158 0 22.7 KB
#5056 RunWaf(args=NestedMap (100)) net6.0 107μs 611ns 4.41μs 0.438 0 0 32.76 KB
#5056 RunWaf(args=NestedMap (100)) netcoreapp3.1 136μs 755ns 4.59μs 0.413 0 0 33.33 KB
#5056 RunWaf(args=NestedMap (100)) net472 193μs 406ns 1.57μs 5.28 0.37 0 33.67 KB
#5056 RunWafTwice(args=NestedMap (100)) net6.0 103μs 22.6ns 87.6ns 0.465 0 0 33.3 KB
#5056 RunWafTwice(args=NestedMap (100)) netcoreapp3.1 140μs 387ns 1.5μs 0.408 0 0 33.86 KB
#5056 RunWafTwice(args=NestedMap (100)) net472 199μs 797ns 3.09μs 5.43 0.388 0 34.23 KB
#5056 RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 160μs 802ns 3.4μs 0.495 0 0 39.1 KB
#5056 RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 197μs 547ns 2.12μs 0.48 0 0 39.63 KB
#5056 RunWafWithAttack(args=Neste(...)tack) [23]) net472 256μs 174ns 673ns 6.38 0.51 0 40.23 KB
#5056 RunWaf(args=NestedMap (20)) net6.0 107μs 47.9ns 173ns 0.457 0 0 32.18 KB
#5056 RunWaf(args=NestedMap (20)) netcoreapp3.1 134μs 741ns 4.5μs 0.444 0 0 32.3 KB
#5056 RunWaf(args=NestedMap (20)) net472 191μs 83.3ns 312ns 5.17 0.369 0 32.63 KB
#5056 RunWafTwice(args=NestedMap (20)) net6.0 111μs 55.4ns 200ns 0.449 0 0 32.72 KB
#5056 RunWafTwice(args=NestedMap (20)) netcoreapp3.1 140μs 685ns 2.74μs 0.429 0 0 32.82 KB
#5056 RunWafTwice(args=NestedMap (20)) net472 194μs 171ns 642ns 5.24 0.388 0 33.19 KB
#5056 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 166μs 152ns 589ns 0.549 0 0 38.53 KB
#5056 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 195μs 1.06μs 6.3μs 0.476 0 0 38.6 KB
#5056 RunWafWithAttack(args=Neste(...)tack) [22]) net472 253μs 140ns 541ns 6.15 0.502 0 39.2 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 175μs 160ns 618ns 0.175 0 0 18.26 KB
master SendRequest netcoreapp3.1 193μs 327ns 1.27μs 0.193 0 0 20.41 KB
master SendRequest net472 0.000101ns 8.48E‑05ns 0.000306ns 0 0 0 0 b
#5056 SendRequest net6.0 173μs 150ns 580ns 0.259 0 0 18.25 KB
#5056 SendRequest netcoreapp3.1 190μs 302ns 1.17μs 0.191 0 0 20.41 KB
#5056 SendRequest net472 0.000478ns 0.000244ns 0.000943ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #5056

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 41.86 KB 41.6 KB -254 B -0.61%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 553μs 2.54μs 9.83μs 0.532 0 0 41.86 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 650μs 1.3μs 5.02μs 0.326 0 0 41.8 KB
master WriteAndFlushEnrichedTraces net472 816μs 2.65μs 10.3μs 8.28 2.48 0.414 53.24 KB
#5056 WriteAndFlushEnrichedTraces net6.0 550μs 342ns 1.28μs 0.553 0 0 41.6 KB
#5056 WriteAndFlushEnrichedTraces netcoreapp3.1 640μs 1.2μs 4.63μs 0.324 0 0 41.6 KB
#5056 WriteAndFlushEnrichedTraces net472 838μs 4.21μs 18.8μs 8.45 2.53 0.422 53.23 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.15μs 0.661ns 2.56ns 0.0103 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.52μs 0.901ns 3.49ns 0.0106 0 0 768 B
master ExecuteNonQuery net472 1.76μs 1.32ns 4.75ns 0.116 0 0 730 B
#5056 ExecuteNonQuery net6.0 1.05μs 0.608ns 2.35ns 0.0104 0 0 768 B
#5056 ExecuteNonQuery netcoreapp3.1 1.46μs 1.2ns 4.64ns 0.0101 0 0 768 B
#5056 ExecuteNonQuery net472 1.82μs 0.792ns 2.86ns 0.115 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.22μs 1.2ns 4.65ns 0.0133 0 0 936 B
master CallElasticsearch netcoreapp3.1 1.55μs 1.12ns 4.34ns 0.0123 0 0 936 B
master CallElasticsearch net472 2.46μs 0.776ns 3.01ns 0.151 0 0 955 B
master CallElasticsearchAsync net6.0 1.33μs 1.05ns 3.93ns 0.0126 0 0 912 B
master CallElasticsearchAsync netcoreapp3.1 1.55μs 0.563ns 2.11ns 0.013 0 0 984 B
master CallElasticsearchAsync net472 2.66μs 0.882ns 3.3ns 0.16 0 0 1.01 KB
#5056 CallElasticsearch net6.0 1.16μs 0.627ns 2.43ns 0.0134 0 0 936 B
#5056 CallElasticsearch netcoreapp3.1 1.47μs 2.68ns 10.4ns 0.0124 0 0 936 B
#5056 CallElasticsearch net472 2.54μs 2.21ns 8.27ns 0.151 0 0 955 B
#5056 CallElasticsearchAsync net6.0 1.25μs 0.572ns 2.22ns 0.0127 0 0 912 B
#5056 CallElasticsearchAsync netcoreapp3.1 1.65μs 1.22ns 4.39ns 0.013 0 0 984 B
#5056 CallElasticsearchAsync net472 2.66μs 2.1ns 8.15ns 0.16 0.00132 0 1.01 KB
Benchmarks.Trace.GraphQLBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5056

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net6.0 1.123 1,221.00 1,371.79

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.22μs 0.712ns 2.66ns 0.0128 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.69μs 0.867ns 3ns 0.012 0 0 912 B
master ExecuteAsync net472 1.88μs 0.687ns 2.48ns 0.138 0 0 875 B
#5056 ExecuteAsync net6.0 1.37μs 0.713ns 2.76ns 0.013 0 0 912 B
#5056 ExecuteAsync netcoreapp3.1 1.69μs 0.589ns 2.12ns 0.0118 0 0 912 B
#5056 ExecuteAsync net472 1.9μs 1.31ns 4.91ns 0.139 0 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 3.9μs 1.3ns 4.85ns 0.0293 0 0 2.1 KB
master SendAsync netcoreapp3.1 4.98μs 2.94ns 11.4ns 0.0347 0 0 2.63 KB
master SendAsync net472 7.76μs 5.47ns 21.2ns 0.522 0 0 3.31 KB
#5056 SendAsync net6.0 4.05μs 6.19ns 23.2ns 0.0282 0 0 2.1 KB
#5056 SendAsync netcoreapp3.1 4.82μs 1.12ns 4.19ns 0.0342 0 0 2.63 KB
#5056 SendAsync net472 7.71μs 4.49ns 17.4ns 0.524 0.00385 0 3.31 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Faster 🎉 More allocations ⚠️

Faster 🎉 in #5056

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net472 1.262 62,600.00 49,600.00 several?
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑netcoreapp3.1 1.179 62,600.00 53,100.00 bimodal

More allocations ⚠️ in #5056

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net472 57.34 KB 73.73 KB 16.38 KB 28.57%

Fewer allocations 🎉 in #5056

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net472 57.34 KB 55.21 KB -2.14 KB -3.72%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) net6.0 60.6μs 894ns 8.71μs 0 0 0 43.44 KB
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 63.3μs 783ns 7.67μs 0 0 0 42.64 KB
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) net472 37.9μs 97.3ns 364ns 0 0 0 57.34 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net6.0 65.6μs 290ns 1.01μs 0 0 0 43.29 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 71.8μs 199ns 717ns 0 0 0 42.64 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net472 62.9μs 197ns 710ns 0 0 0 57.34 KB
#5056 StringConcatBenchmark(parameters=Syste(...)ring] [48]) net6.0 51.7μs 199ns 716ns 0 0 0 43.44 KB
#5056 StringConcatBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 53μs 246ns 921ns 0 0 0 42.64 KB
#5056 StringConcatBenchmark(parameters=Syste(...)ring] [48]) net472 37.9μs 144ns 519ns 0 0 0 55.21 KB
#5056 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net6.0 65.6μs 189ns 680ns 0 0 0 43.29 KB
#5056 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 110μs 3.78μs 37.4μs 0 0 0 42.64 KB
#5056 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net472 54.5μs 840ns 8.31μs 0 0 0 73.73 KB
Benchmarks.Trace.ILoggerBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5056

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog‑net6.0 1.145 1,611.46 1,407.85

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.61μs 0.662ns 2.48ns 0.0219 0 0 1.57 KB
master EnrichedLog netcoreapp3.1 2.3μs 0.785ns 2.83ns 0.0207 0 0 1.57 KB
master EnrichedLog net472 2.61μs 2.24ns 8.07ns 0.238 0 0 1.5 KB
#5056 EnrichedLog net6.0 1.41μs 0.857ns 3.21ns 0.022 0 0 1.57 KB
#5056 EnrichedLog netcoreapp3.1 2.17μs 1.4ns 5.23ns 0.021 0 0 1.57 KB
#5056 EnrichedLog net472 2.6μs 2.38ns 9.21ns 0.238 0 0 1.5 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 87.2ns 326ns 0.0566 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 118μs 91.4ns 342ns 0 0 0 4.21 KB
master EnrichedLog net472 148μs 37.1ns 139ns 0.665 0.222 0 4.39 KB
#5056 EnrichedLog net6.0 114μs 332ns 1.28μs 0.0576 0 0 4.21 KB
#5056 EnrichedLog netcoreapp3.1 119μs 153ns 594ns 0 0 0 4.21 KB
#5056 EnrichedLog net472 148μs 58ns 217ns 0.664 0.221 0 4.39 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3μs 0.952ns 3.69ns 0.0299 0 0 2.13 KB
master EnrichedLog netcoreapp3.1 4.11μs 1.64ns 6.37ns 0.0289 0 0 2.13 KB
master EnrichedLog net472 4.91μs 1.84ns 6.88ns 0.309 0 0 1.95 KB
#5056 EnrichedLog net6.0 3.07μs 1.49ns 5.78ns 0.0291 0 0 2.13 KB
#5056 EnrichedLog netcoreapp3.1 4.06μs 1.65ns 6.39ns 0.0284 0 0 2.13 KB
#5056 EnrichedLog net472 4.91μs 1.19ns 4.29ns 0.309 0 0 1.95 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.35μs 1.76ns 6.82ns 0.0154 0 0 1.1 KB
master SendReceive netcoreapp3.1 1.81μs 0.894ns 3.35ns 0.0145 0 0 1.1 KB
master SendReceive net472 2.06μs 3.21ns 12.4ns 0.177 0 0 1.12 KB
#5056 SendReceive net6.0 1.37μs 0.574ns 2.22ns 0.0151 0 0 1.1 KB
#5056 SendReceive netcoreapp3.1 1.86μs 0.694ns 2.6ns 0.0148 0 0 1.1 KB
#5056 SendReceive net472 2.14μs 1.38ns 5.15ns 0.177 0 0 1.12 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.66μs 1.85ns 6.93ns 0.0214 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.99μs 2.82ns 10.9ns 0.0199 0 0 1.58 KB
master EnrichedLog net472 4.3μs 1.86ns 7.21ns 0.311 0 0 1.97 KB
#5056 EnrichedLog net6.0 2.93μs 0.65ns 2.52ns 0.0212 0 0 1.53 KB
#5056 EnrichedLog netcoreapp3.1 4.01μs 1.81ns 6.77ns 0.02 0 0 1.58 KB
#5056 EnrichedLog net472 4.34μs 1.87ns 6.99ns 0.312 0 0 1.97 KB
Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5056

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1 1.138 797.19 700.57

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 471ns 0.189ns 0.733ns 0.00738 0 0 536 B
master StartFinishSpan netcoreapp3.1 797ns 0.319ns 1.24ns 0.00718 0 0 536 B
master StartFinishSpan net472 762ns 0.25ns 0.967ns 0.0852 0 0 538 B
master StartFinishScope net6.0 629ns 0.117ns 0.42ns 0.00914 0 0 656 B
master StartFinishScope netcoreapp3.1 880ns 0.208ns 0.806ns 0.00882 0 0 656 B
master StartFinishScope net472 950ns 0.746ns 2.89ns 0.0978 0 0 618 B
#5056 StartFinishSpan net6.0 452ns 0.152ns 0.588ns 0.00759 0 0 536 B
#5056 StartFinishSpan netcoreapp3.1 701ns 0.383ns 1.38ns 0.00716 0 0 536 B
#5056 StartFinishSpan net472 764ns 0.332ns 1.29ns 0.0851 0 0 538 B
#5056 StartFinishScope net6.0 672ns 0.253ns 0.948ns 0.00913 0 0 656 B
#5056 StartFinishScope netcoreapp3.1 806ns 0.265ns 1.03ns 0.00893 0 0 656 B
#5056 StartFinishScope net472 960ns 0.612ns 2.37ns 0.0982 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5056

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net472 1.147 994.88 1,140.95

Faster 🎉 in #5056

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 1.139 716.86 629.17

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 717ns 0.56ns 2.17ns 0.00906 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 982ns 0.57ns 2.21ns 0.00859 0 0 656 B
master RunOnMethodBegin net472 995ns 0.308ns 1.19ns 0.0977 0 0 618 B
#5056 RunOnMethodBegin net6.0 629ns 0.117ns 0.423ns 0.00909 0 0 656 B
#5056 RunOnMethodBegin netcoreapp3.1 937ns 0.549ns 2.12ns 0.00882 0 0 656 B
#5056 RunOnMethodBegin net472 1.14μs 0.666ns 2.58ns 0.098 0 0 618 B

pierotibou
pierotibou approved these changes Jan 17, 2024
View reviewed changes
Copy link
Contributor

@pierotibou pierotibou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latter one means we weren't actually running all those tests

If I have seen correctly, we were not running some serverless tests as well recently. Could we have a way to list tests that aren't run somehow and monitor this? It's worrying we may not be running the tests we think we test

@andrewlock andrewlock merged commit 12b5258 into master Jan 18, 2024
@andrewlock andrewlock deleted the andrew/test-fixes branch January 18, 2024 10:49
@github-actions github-actions bot added this to the vNext milestone Jan 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zacharycmontoya zacharycmontoya zacharycmontoya approved these changes

@pierotibou pierotibou pierotibou approved these changes

Assignees
No one assigned
Labels
area:tests unit tests, integration tests
Projects
None yet
Milestone
2.46.0
Development

Successfully merging this pull request may close these issues.
3 participants
@andrewlock @zacharycmontoya @pierotibou