[Internal] Binary Encoding: Fixes DateTime Parsing Issue with Trailing Zeros in the Milli-Seconds Precision.

[aavasthy]

Pull Request Template

Background:

During some of the internal validation, it was identified that when the cosmos .net SDK is initialized with binary encoding, and a date time field is persisted into a database, the value is getting stored with trailing zeros. To understand this better, please see the following steps to repro the issue.

Goal is to identify the serialization gap here and fix it in the .NET SDK.

Steps to Repro:

• Cosmos Client Creation:

        Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True");
        string databaseName = "binary-encoding-db";
        string containerName = "binary-encoding-container";
        CosmosClientOptions clientOptions = new()
        {
            ConnectionMode = ConnectionMode.Direct,
            RequestTimeout = TimeSpan.FromSeconds(5000),
        };

        CosmosClient client = new CosmosClient(connectionString,
            clientOptions);

• Item Creation:

      string[] dateStrings = {
          "12/25/2023",
          "2023-12-25",
          "12-25-2023",
          "25.12.2023",
          "25/12/2023",
          "Dec 25, 2023",
          "Dec 25 2023",
          "2023-12-25T10:00:00",
          "2023-12-25T10:00:00.123",
          "12/25/2023 10:00 AM",
          "12/25/2023 10:00:00 AM",
          "12/25/2023 10:00:00.123 AM"
      };

      // Define the supported formats
      string[] formats = {
          "MM/dd/yyyy",
          "yyyy-MM-dd",
          "MM-dd-yyyy",
          "dd.MM.yyyy",
          "dd/MM/yyyy",
          "MMM dd, yyyy",
          "MMM dd yyyy",
          "yyyy-MM-ddTHH:mm:ss",
          "yyyy-MM-ddTHH:mm:ss.fff",
          "MM/dd/yyyy hh:mm tt",
          "MM/dd/yyyy hh:mm:ss tt",
          "MM/dd/yyyy hh:mm:ss.fff tt"
      };

      int idx = 0;
      DateTime[] dateTimes = new DateTime[dateStrings.Length];
      // Try to parse each date string
      foreach (string dateString in dateStrings)
      {
          try
          {
              // Use TryParseExact to handle multiple formats
              DateTime parsedDate = DateTime.ParseExact(dateString, formats, CultureInfo.InvariantCulture, DateTimeStyles.None);
              dateTimes[idx++] = parsedDate;
              // Display the parsed date
          }
          catch (FormatException)
          {
              Console.WriteLine($"Failed to parse: {dateString}");
          }
      }


      Comment comment = new Comment(
          Guid.NewGuid().ToString(),
          "pk", random.Next().ToString(),
          "[email protected]",
          "This document is intended for binary encoding demo.",
          createdAt: DateTime.UtcNow,
          modifiedAt: DateTime.Parse("2025-03-26T20:22:20Z"),
          datetimes: dateTimes);

      ItemResponse<Comment> writeResponse = await container.CreateItemAsync<Comment>(
          item: comment,
          partitionKey: new Cosmos.PartitionKey(comment.pk),
          requestOptions: requestOptions
      );

      Console.WriteLine(writeResponse .Diagnostics);

• Sample Data Saved in Container. Please see below the comparison results, when binary encoding is Disabled vs Enabled:

Analysis:

Initially, it appears that the current binary DateTime parsing logic, in the CosmosDBToNewtonsoftWriter uses the round-trip date/time pattern, which is responsible for adding the trailing zeros, if the milli seconds precision is missing:

    /// <summary>
    /// Writes a <see cref="DateTime"/> value.
    /// </summary>
    /// <param name="value">The <see cref="DateTime"/> value to write.</param>
    public override void WriteValue(DateTime value)
    {
        this.WriteValue(value.ToString("O"));
    }

Fix:

The PR is fixing the above issue. The built-in DateTime format "o"/ "O" is comparable to the custom format of: "yyyy'-'MM'-'dd'T'HH':'mm':'ss'.'fffffffK". In order to remove the trailing zeros from the milli-second precision, we replace the lower-case f's with upper case ones, i.e. "yyyy'-'MM'-'dd'T'HH':'mm':'ss'.'FFFFFFFK" custom date-time format takes care of fixing it.

Type of change

• Bug fix (non-breaking change which fixes an issue)

Closing issues

To automatically close an issue: closes #5213

[sboshra]

[FabianMeiswinkel]

LGTM - Thanks!