[Test Package Versions Bump] Updating package versions #7515

dd-octo-sts · 2025-09-14T00:30:48Z

Updates the package versions for integration tests.

dd-trace-dotnet-ci-bot · 2025-09-14T01:10:47Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7515) - mean (80ms)  : crit, 75, 85
     .   : crit, milestone, 80,
    master - mean (72ms)  : 71, 73
     .   : milestone, 72,

    section Baseline
    This PR (7515) - mean (75ms)  : 71, 78
     .   : milestone, 75,
    master - mean (68ms)  : 66, 71
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (7515) - mean (1,109ms)  : crit, 1057, 1162
     .   : crit, milestone, 1109,
    master - mean (1,047ms)  : 997, 1097
     .   : milestone, 1047,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7515) - mean (118ms)  : crit, 114, 122
     .   : crit, milestone, 118,
    master - mean (106ms)  : 105, 107
     .   : milestone, 106,

    section Baseline
    This PR (7515) - mean (117ms)  : 112, 122
     .   : milestone, 117,
    master - mean (106ms)  : 103, 108
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (7515) - mean (789ms)  : crit, 766, 813
     .   : crit, milestone, 789,
    master - mean (741ms)  : 719, 764
     .   : milestone, 741,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7515) - mean (112ms)  : crit, 108, 116
     .   : crit, milestone, 112,
    master - mean (94ms)  : 93, 95
     .   : milestone, 94,

    section Baseline
    This PR (7515) - mean (110ms)  : 105, 116
     .   : milestone, 110,
    master - mean (94ms)  : 91, 96
     .   : milestone, 94,

    section CallTarget+Inlining+NGEN
    This PR (7515) - mean (822ms)  : crit, 760, 884
     .   : crit, milestone, 822,
    master - mean (701ms)  : 684, 719
     .   : milestone, 701,

gantt
    title Execution time (ms) FakeDbCommand (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7515) - mean (104ms)  : crit, 100, 108
     .   : crit, milestone, 104,
    master - mean (92ms)  : 91, 94
     .   : milestone, 92,

    section Baseline
    This PR (7515) - mean (103ms)  : 96, 110
     .   : milestone, 103,
    master - mean (92ms)  : 90, 94
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (7515) - mean (707ms)  : crit, 686, 728
     .   : crit, milestone, 707,
    master - mean (666ms)  : 648, 683
     .   : milestone, 666,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7515) - mean (209ms)  : 203, 215
     .   : milestone, 209,
    master - mean (199ms)  : 195, 202
     .   : milestone, 199,

    section Baseline
    This PR (7515) - mean (202ms)  : 190, 215
     .   : milestone, 202,
    master - mean (196ms)  : 191, 201
     .   : milestone, 196,

    section CallTarget+Inlining+NGEN
    This PR (7515) - mean (1,194ms)  : 1129, 1259
     .   : milestone, 1194,
    master - mean (1,180ms)  : 1114, 1246
     .   : milestone, 1180,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7515) - mean (288ms)  : 279, 297
     .   : milestone, 288,
    master - mean (282ms)  : 276, 287
     .   : milestone, 282,

    section Baseline
    This PR (7515) - mean (286ms)  : 278, 295
     .   : milestone, 286,
    master - mean (282ms)  : 277, 286
     .   : milestone, 282,

    section CallTarget+Inlining+NGEN
    This PR (7515) - mean (959ms)  : 920, 998
     .   : milestone, 959,
    master - mean (943ms)  : 905, 982
     .   : milestone, 943,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7515) - mean (289ms)  : 280, 299
     .   : milestone, 289,
    master - mean (275ms)  : 270, 280
     .   : milestone, 275,

    section Baseline
    This PR (7515) - mean (290ms)  : 283, 297
     .   : milestone, 290,
    master - mean (276ms)  : 270, 281
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (7515) - mean (1,009ms)  : crit, 970, 1047
     .   : crit, milestone, 1009,
    master - mean (931ms)  : 891, 971
     .   : milestone, 931,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7515) - mean (279ms)  : 269, 289
     .   : milestone, 279,
    master - mean (276ms)  : 269, 282
     .   : milestone, 276,

    section Baseline
    This PR (7515) - mean (275ms)  : 268, 281
     .   : milestone, 275,
    master - mean (273ms)  : 268, 278
     .   : milestone, 273,

    section CallTarget+Inlining+NGEN
    This PR (7515) - mean (861ms)  : 839, 882
     .   : milestone, 861,
    master - mean (863ms)  : 843, 883
     .   : milestone, 863,

andrewlock · 2025-09-15T08:15:03Z

tracer/dependabot/integrations/Datadog.Dependabot.MongoDB.Driver.csproj

+    <!-- Latest package https://www.nuget.org/packages/MongoDB.Driver/3.5.0 -->
+    <PackageReference Include="MongoDB.Driver" Version="3.5.0" />


This is failing because it requires a newer version of the mongodb itself. That means we definitely need to update the mongodb image (and so update all the VMs), but it may also mean that we actually need to run both old and new versions of mongodb, and split which one we talk to based on the package version 🙁

01:25:04 [DBG] Unhandled exception. MongoDB.Driver.MongoIncompatibleDriverException: Server at mongo:27017 reports wire version 7, but this version of the driver requires at least 8 (MongoDB 4.2.0).

https://datadoghq.atlassian.net/browse/LANGPLAT-860 -> work item to bump the docker image and update the VMs

I will go and cap the version to less than 3.5.0 for CI and then test this out locally I have that here
https://datadoghq.atlassian.net/browse/LANGPLAT-861

Capped it in 787ff19

Will need to test locally to make sure it works and prepare a PR for that to get it updated.

bouwkast · 2025-10-01T17:03:09Z

tracer/dependabot/integrations/Datadog.Dependabot.MSTest.TestAdapter.csproj

-    <!-- Latest package https://www.nuget.org/packages/MSTest.TestAdapter/3.10.4 -->
-    <PackageReference Include="MSTest.TestAdapter" Version="3.10.4" />
+    <!-- Latest package https://www.nuget.org/packages/MSTest.TestAdapter/3.11.0 -->
+    <PackageReference Include="MSTest.TestAdapter" Version="3.11.0" />


This appears to not work 😢

Okay so 3.10.4 works, 3.10.5 has some failures, 3.11.0 has a lot of failures

Just re-ran GeneratePackageVersions I tried to limit it to just MongoDB.Driver, but that didn't work.

Tests are failing in 3.11.0 which just released

bouwkast

Approving

We have some follow ups here though

Test against MongoDB.Driver 3.5.0 locally, fix if necessary
PR to get support for that new docker version
Fix the MSTest.TestFramework instrumentation

Dismissing as I've pinned the version to below 3.5.0 for MongoDB.Driver and have scheduled work to remediate

Updates the package versions for integration tests. --------- Co-authored-by: andrewlock <[email protected]> Co-authored-by: Steven Bouwkamp <[email protected]>

dd-octo-sts bot requested review from a team as code owners September 14, 2025 00:30

dd-octo-sts bot added area:dependabot dependabot updates area:test-apps apps used to test integrations dependencies Pull requests that update a dependency file labels Sep 14, 2025

This comment has been minimized.

Sign in to view

andrewlock previously requested changes Sep 15, 2025

View reviewed changes

dd-octo-sts bot force-pushed the bot/test-package-versions-bump branch from e645e2c to 4b239dc Compare September 21, 2025 00:27

dd-octo-sts bot force-pushed the bot/test-package-versions-bump branch from 4b239dc to 12b0247 Compare September 28, 2025 00:30

bouwkast reviewed Oct 1, 2025

View reviewed changes

andrewlock and others added 4 commits October 1, 2025 16:39

[Test Package Versions Bump]

ea7675f

Pin test to less than 3.5.0 of MongoDB.Driver

986916b

Just re-ran GeneratePackageVersions I tried to limit it to just MongoDB.Driver, but that didn't work.

Downgrade MSTest.TestFramework to 3.10.5

ef27823

Tests are failing in 3.11.0 which just released

Downgrade MSTest.TestFramework to 3.10.4

802957d

bouwkast force-pushed the bot/test-package-versions-bump branch from 1d83a2a to 802957d Compare October 1, 2025 20:40

Pin MSTest.TestFramework to 3.10.4 for testing

aaa929d

bouwkast force-pushed the bot/test-package-versions-bump branch from 5377599 to aaa929d Compare October 3, 2025 12:51

bouwkast requested a review from andrewlock October 3, 2025 13:47

bouwkast approved these changes Oct 3, 2025

View reviewed changes

bouwkast merged commit b735d73 into master Oct 3, 2025
153 checks passed

bouwkast deleted the bot/test-package-versions-bump branch October 3, 2025 15:20

github-actions bot added this to the vNext-v3 milestone Oct 3, 2025

bouwkast mentioned this pull request Oct 14, 2025

[Test Package Versions Bump] Updating package versions #7644

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Test Package Versions Bump] Updating package versions #7515

[Test Package Versions Bump] Updating package versions #7515

Uh oh!

dd-octo-sts bot commented Sep 14, 2025

Uh oh!

dd-trace-dotnet-ci-bot bot commented Sep 14, 2025 •

edited

Loading

Uh oh!

This comment has been minimized.

andrewlock Sep 15, 2025

Uh oh!

bouwkast Sep 15, 2025

Uh oh!

bouwkast Sep 30, 2025

Uh oh!

bouwkast Oct 1, 2025

Uh oh!

bouwkast Oct 1, 2025

Uh oh!

bouwkast Oct 1, 2025

Uh oh!

bouwkast left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		<!-- Latest package https://www.nuget.org/packages/MongoDB.Driver/3.5.0 -->
		<PackageReference Include="MongoDB.Driver" Version="3.5.0" />

Uh oh!

[Test Package Versions Bump] Updating package versions #7515

[Test Package Versions Bump] Updating package versions #7515

Uh oh!

Conversation

dd-octo-sts bot commented Sep 14, 2025

Uh oh!

dd-trace-dotnet-ci-bot bot commented Sep 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Execution-Time Benchmarks Report ⏱️

Uh oh!

This comment has been minimized.

andrewlock Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

bouwkast Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

bouwkast Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

bouwkast Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

bouwkast Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

bouwkast Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

bouwkast left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dd-trace-dotnet-ci-bot bot commented Sep 14, 2025 •

edited

Loading