Improve and simplify the microbenchmarks CI setup #7571
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing the following branches/commits: Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat X
axisFormat %s
todayMarker off
section Bailout
This PR (7571) - mean (72ms) : 71, 73
. : milestone, 72,
master - mean (72ms) : 71, 73
. : milestone, 72,
section Baseline
This PR (7571) - mean (68ms) : 66, 71
. : milestone, 68,
master - mean (68ms) : 65, 71
. : milestone, 68,
section CallTarget+Inlining+NGEN
This PR (7571) - mean (1,056ms) : 992, 1121
. : milestone, 1056,
master - mean (1,044ms) : 1011, 1076
. : milestone, 1044,
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Bailout
This PR (7571) - mean (107ms) : 106, 108
. : milestone, 107,
master - mean (106ms) : 105, 108
. : milestone, 106,
section Baseline
This PR (7571) - mean (106ms) : 104, 108
. : milestone, 106,
master - mean (106ms) : 104, 108
. : milestone, 106,
section CallTarget+Inlining+NGEN
This PR (7571) - mean (743ms) : 724, 762
. : milestone, 743,
master - mean (746ms) : 727, 764
. : milestone, 746,
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Bailout
This PR (7571) - mean (101ms) : 100, 102
. : milestone, 101,
master - mean (100ms) : 100, 101
. : milestone, 100,
section Baseline
This PR (7571) - mean (100ms) : 98, 102
. : milestone, 100,
master - mean (100ms) : 98, 103
. : milestone, 100,
section CallTarget+Inlining+NGEN
This PR (7571) - mean (770ms) : 724, 816
. : milestone, 770,
master - mean (775ms) : 723, 827
. : milestone, 775,
gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat X
axisFormat %s
todayMarker off
section Bailout
This PR (7571) - mean (93ms) : 91, 94
. : milestone, 93,
master - mean (93ms) : 92, 94
. : milestone, 93,
section Baseline
This PR (7571) - mean (92ms) : 90, 95
. : milestone, 92,
master - mean (92ms) : 89, 95
. : milestone, 92,
section CallTarget+Inlining+NGEN
This PR (7571) - mean (660ms) : 646, 673
. : milestone, 660,
master - mean (664ms) : 649, 678
. : milestone, 664,
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat X
axisFormat %s
todayMarker off
section Bailout
This PR (7571) - mean (208ms) : 195, 222
. : milestone, 208,
master - mean (200ms) : 197, 204
. : milestone, 200,
section Baseline
This PR (7571) - mean (203ms) : 192, 214
. : milestone, 203,
master - mean (200ms) : 189, 211
. : milestone, 200,
section CallTarget+Inlining+NGEN
This PR (7571) - mean (1,211ms) : 1144, 1277
. : milestone, 1211,
master - mean (1,199ms) : 1130, 1268
. : milestone, 1199,
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Bailout
This PR (7571) - mean (295ms) : 279, 310
. : milestone, 295,
master - mean (286ms) : 277, 296
. : milestone, 286,
section Baseline
This PR (7571) - mean (293ms) : 276, 309
. : milestone, 293,
master - mean (283ms) : 275, 291
. : milestone, 283,
section CallTarget+Inlining+NGEN
This PR (7571) - mean (977ms) : 930, 1025
. : milestone, 977,
master - mean (950ms) : 902, 998
. : milestone, 950,
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Bailout
This PR (7571) - mean (296ms) : 282, 309
. : milestone, 296,
master - mean (289ms) : 277, 301
. : milestone, 289,
section Baseline
This PR (7571) - mean (297ms) : 279, 315
. : milestone, 297,
master - mean (288ms) : 280, 296
. : milestone, 288,
section CallTarget+Inlining+NGEN
This PR (7571) - mean (1,030ms) : 974, 1085
. : milestone, 1030,
master - mean (1,011ms) : 953, 1069
. : milestone, 1011,
gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat X
axisFormat %s
todayMarker off
section Bailout
This PR (7571) - mean (285ms) : 268, 301
. : milestone, 285,
master - mean (276ms) : 267, 285
. : milestone, 276,
section Baseline
This PR (7571) - mean (286ms) : 267, 304
. : milestone, 286,
master - mean (278ms) : 269, 287
. : milestone, 278,
section CallTarget+Inlining+NGEN
This PR (7571) - mean (921ms) : 820, 1021
. : milestone, 921,
master - mean (875ms) : 847, 902
. : milestone, 875,
|
- Renames the AMI cretion job for `build-dd-trace-dotnet-microbenchmarks-ami`. - Uses `dd-octo-sts` for secure GitHub token generation. - Simplifies and reduces the number of CI variables, standardizing on the `BP_INFRA_*` prefix. - Makes instance cleanup conditional on a `CLEANUP` variable. This allows us to SSH/RDP into instances after benchmarks are run, if necessary.
igoragoli
force-pushed
the
augusto/dd-trace-dotnet-micro-bp-ui
branch
from
e85f962 to
34ec104
Compare
igoragoli
changed the title
BenchmarksBenchmarks Report for benchmark platform 🐌Benchmarks for #7571 compared to master:
The following thresholds were used for comparing the benchmark speeds:
Allocation changes below 0.5% are ignored. Benchmark detailsBenchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.CharSliceBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️Raw results
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting
NachoEchevarria
approved these changes
Oct 1, 2025
igoragoli
added a commit
that referenced
this pull request
Oct 2, 2025
## Summary of changes Reverts #7571 since it introduces a bug preventing microbenchmarks to be run. ## Reason for change ## Implementation details ## Test coverage ## Other details <!-- Fixes #{issue} --> <!--⚠️ Note: Where possible, please obtain 2 approvals prior to merging. Unless CODEOWNERS specifies otherwise, for external teams it is typically best to have one review from a team member, and one review from apm-dotnet. Trivial changes do not require 2 reviews. MergeQueue is NOT enabled in this repository. If you have write access to the repo, the PR has 1-2 approvals (see above), and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #apm-dotnet channel in Slack. -->
1 task
bouwkast
added
the
area:builds
igoragoli
added a commit
that referenced
this pull request
Oct 8, 2025
- [x] Revert [a7749d8](a7749d8) before merging. ## Summary of changes - Restore changes from the now-reverted #7571 PR. - Fixes the dd-octo-sts GitHub token setup by creating a token targeting benchmarking-platform instead of dd-trace-dotnet when running benchmarks. - Corresponding dd-octo-sts policy on benchmarking-platform: https://github.com/DataDog/benchmarking-platform/blob/main/.github/chainguard/gitlab.github-access.read-contents.sts.yaml ## Reason for change - We couldn't clone benchmarking-platform from ephemeral benchmarking instances with the dd-octo-sts setup introduced on #7571. ## Implementation details ## Test coverage Test run on the CI: https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-dotnet/-/jobs/1161521230 ## Other details <!-- Fixes #{issue} --> <!--⚠️ Note: Where possible, please obtain 2 approvals prior to merging. Unless CODEOWNERS specifies otherwise, for external teams it is typically best to have one review from a team member, and one review from apm-dotnet. Trivial changes do not require 2 reviews. MergeQueue is NOT enabled in this repository. If you have write access to the repo, the PR has 1-2 approvals (see above), and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #apm-dotnet channel in Slack. -->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
microbenchmarks.ymlbefore merging.Summary of changes
build-dd-trace-dotnet-microbenchmarks-ami.dd-octo-stsfor generating GitHub tokens.BP_INFRA_*prefix.CLEANUPvariable. This allows us to SSH/RDP into instances after benchmarks are run, if necessary.Related changes on benchmarking-platform: refactor: simplify dd-trace-dotnet microbenchmarks
Reason for change
https://datadoghq.atlassian.net/browse/APMSP-2282 and https://datadoghq.atlassian.net/browse/APMSP-1908.
Implementation details
Test coverage
Benchmark run on the CI: https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-dotnet/-/jobs/1154611844
Other details