Trace: Fixes thread safety issue in Trace class causing high CPU usage and InvalidOperationException #5361
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…concurrent access Co-authored-by: kirankumarkolli <[email protected]>
…tion points Co-authored-by: kirankumarkolli <[email protected]>
…tions Co-authored-by: kirankumarkolli <[email protected]>
Since isBeingWalked flag prevents modifications during enumeration, snapshot methods are no longer needed. The trace system now uses direct enumeration with walking state protection instead of creating copies. Co-authored-by: kirankumarkolli <[email protected]>
…mosTraceDiagnostics Co-authored-by: kirankumarkolli <[email protected]>
Co-authored-by: kirankumarkolli <[email protected]>
…move parameter Co-authored-by: kirankumarkolli <[email protected]>
… adjust assignment order Co-authored-by: kirankumarkolli <[email protected]>
…as started - but instead just operate on cloned snapshots
FabianMeiswinkel
requested review from
khdang,
sboshra,
adityasa,
neildsh,
kirankumarkolli,
kirillg and
Pilchie
as code owners
1 task
kirankumarkolli
approved these changes
Aug 19, 2025
aavasthy
approved these changes
Aug 19, 2025
This was referenced Aug 27, 2025
aavasthy
pushed a commit
that referenced
this pull request
Aug 27, 2025
…e and InvalidOperationException (#5361) # Pull Request Template ## Description When cross-region hedging is enabled in the Azure Cosmos DB .NET SDK, users experience high CPU usage or InvalidOperationExceptions ue to thread safety issues when accessing dictionaries in the Trace class. The issue occurs in the AddOrUpdateDatum method where multiple parallel threads attempt to update the same dictionary without proper synchronization. ### Design considerations The approach chosen is motivated by: - the need to maintain the order of TraceDatum nodes - the laternative of switching to a ConcurrentDictionary instead off keeping Dictonary and making it thread-safe would result in reordering of trace datum nodes - non-functional optimization for the normal request flow where materialization o fthe diagnostics only strats after request processing completes (and no more new trace datum nodes/children) are added to a trace. There are a few edge cases like cross regional retries and timeouts where trace datum nodes could still be added after starting to materialize the trace. The current design marks the begin of materialization and would afterwards create new snapshots for every mutation. This allows making sure that Trace.Data or Trace.Children oeprate on a snapshot that is immutable (thread-safe) - but between multiple calls to ITrace.Data/Children different isntances could be returned - and they might have different content. ## Type of change Please delete options that are not relevant. - [x] Bug fix (non-breaking change which fixes an issue) - [] New feature (non-breaking change which adds functionality) - [] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [] This change requires a documentation update ## Closing issues To automatically close an issue: closes #IssueNumber --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: kirankumarkolli <[email protected]> Co-authored-by: Kiran Kumar Kolli <[email protected]>
This was referenced Aug 29, 2025
This was referenced Sep 8, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Template
Description
When cross-region hedging is enabled in the Azure Cosmos DB .NET SDK, users experience high CPU usage or InvalidOperationExceptions ue to thread safety issues when accessing dictionaries in the Trace class. The issue occurs in the AddOrUpdateDatum method where multiple parallel threads attempt to update the same dictionary without proper synchronization.
Design considerations
The approach chosen is motivated by:
Type of change
Please delete options that are not relevant.
Closing issues
To automatically close an issue: closes #IssueNumber