Trace: Fixes thread safety issue in Trace class causing high CPU usage and InvalidOperationException

[FabianMeiswinkel]

Pull Request Template

Description

When cross-region hedging is enabled in the Azure Cosmos DB .NET SDK, users experience high CPU usage or InvalidOperationExceptions ue to thread safety issues when accessing dictionaries in the Trace class. The issue occurs in the AddOrUpdateDatum method where multiple parallel threads attempt to update the same dictionary without proper synchronization.

Design considerations

The approach chosen is motivated by:

• the need to maintain the order of TraceDatum nodes - the laternative of switching to a ConcurrentDictionary instead off keeping Dictonary and making it thread-safe would result in reordering of trace datum nodes
• non-functional optimization for the normal request flow where materialization o fthe diagnostics only strats after request processing completes (and no more new trace datum nodes/children) are added to a trace. There are a few edge cases like cross regional retries and timeouts where trace datum nodes could still be added after starting to materialize the trace. The current design marks the begin of materialization and would afterwards create new snapshots for every mutation. This allows making sure that Trace.Data or Trace.Children oeprate on a snapshot that is immutable (thread-safe) - but between multiple calls to ITrace.Data/Children different isntances could be returned - and they might have different content.

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• [] New feature (non-breaking change which adds functionality)
• [] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [] This change requires a documentation update

Closing issues

To automatically close an issue: closes #IssueNumber