Support for ChatOptions.ResponseFormat in AWSSDK.Extensions.Bedrock.MEAI

[AlexDaines]

Add ChatOptions.ResponseFormat support for Bedrock MEAI

Description

Implements support for ChatOptions.ResponseFormat in the AWSSDK.Extensions.Bedrock.MEAI implementation of IChatClient. When ResponseFormat is set to Json or ForJsonSchema, the client now uses Bedrock's tool mechanism to enforce structured JSON responses from models.

Implementation approach:

• Creates a synthetic tool called "generate_response" with the provided JSON schema
• Forces model to use this tool via toolChoice
• Extracts JSON response from the tool use output
• Converts Bedrock Document objects to standard JSON

Key behavior:

• ResponseFormat.Json: Requests JSON with generic object schema
• ResponseFormat.ForJsonSchema: Requests JSON conforming to custom schema
• ResponseFormat.Text: No changes to request (default behavior)
• Throws ArgumentException if ResponseFormat is used with user-provided tools (mutual exclusivity)
• Throws NotSupportedException for streaming requests (Bedrock limitation)

Motivation and Context

Closes #3911

Users need consistent behavior when using IChatClient across different AI providers. Currently, the Bedrock implementation ignores ChatOptions.ResponseFormat, making it impossible to request structured responses through the standardized Microsoft.Extensions.AI interface. This prevents Bedrock from being a drop-in replacement for other providers in structured data workflows.

Testing

• Added 2 core unit tests covering request creation with schemas and response extraction
• Created local sample application demonstrating the feature with real Bedrock API calls
• Verified JSON responses are correctly structured and parsed
• Confirmed error handling for invalid configurations (tools + ResponseFormat, streaming)

Test coverage:

• ResponseFormat_Json_WithSchema_CreatesSyntheticToolWithCorrectSchema: Validates synthetic tool creation with custom schema
• ResponseFormat_Json_ModelReturnsToolUse_ExtractsJsonCorrectly: Validates JSON extraction from tool use responses

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist

• My code follows the code style of this project
• My change requires a change to the documentation
• I have updated the documentation accordingly
• I have read the README document
• I have added tests to cover my changes
• All new and existing tests passed

License

• I confirm that this pull request can be released under the Apache 2 license

[Copilot]

Pull Request Overview

This PR adds ResponseFormat support to the AWS Bedrock ChatClient for Microsoft.Extensions.AI, enabling structured JSON output from Bedrock models. The implementation uses Bedrock's tool mechanism with a synthetic tool to enforce structured responses, requiring models with ToolChoice support (Claude 3+ and Mistral Large).

Key changes:

• Implemented ResponseFormat handling via synthetic tool creation that forces models to return structured JSON
• Added error handling for unsupported models and missing structured responses
• Added Document-to-JSON conversion utilities for extracting structured content from tool use responses

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
extensions/src/AWSSDK.Extensions.Bedrock.MEAI/BedrockChatClient.cs	Core implementation of ResponseFormat support including synthetic tool creation, error handling for unsupported models, Document-to-JSON conversion, and validation that ResponseFormat conflicts with user-provided tools
extensions/test/BedrockMEAITests/BedrockChatClientTests.cs	Added MockBedrockRuntime test infrastructure and two tests validating schema conversion and JSON extraction from tool use responses

[Copilot]

This PR appears to be missing a DevConfig file, which is required according to the CONTRIBUTING.md guidelines. This change adds new functionality (ResponseFormat support) to the Bedrock MEAI extension, which constitutes a minor version bump. A DevConfig file should be created in the generator/.DevConfigs directory with appropriate changelog messages and version type specification.

[Copilot]

The depth limit check for preventing stack overflow in recursive JSON conversion only protects against excessive nesting but doesn't protect against actual circular references in the Document structure. If a Document contains a circular reference (e.g., a dictionary that references itself), this will still cause infinite recursion until the depth limit is reached.

Consider tracking visited Document instances to detect actual circular references earlier, or document that Document structures are expected to be acyclic.

[Copilot]

The error detection logic uses multiple string checks for variations of "toolChoice", but this approach is fragile and could produce false positives. For example, if an error message contains "toolChoice" in a different context (e.g., "Invalid parameter value, not related to toolChoice functionality"), it would incorrectly match.

Consider either:

1. Using a more specific error code if one exists for this scenario
2. Using a regular expression with word boundaries to ensure "toolChoice" is a distinct term
3. Checking for more specific error message patterns that uniquely identify this error

[GarrettBeatty]

can you explain more a little more in the PR why we need make this synthetic tool and what not

[GarrettBeatty]

why are these docs deleted

[GarrettBeatty]

just wondering why this is

[GarrettBeatty]

all of this stuff i would be surprised if it doesnt exist already in a jsonutils or utils file. either way it shouldnt be in this class

[peterrsongg]

Agree with garret, it should live in a utils class at the least

[dscpinheiro]

I haven't looked at the entire PR, but we've had requests in the past to make the Document class interop better with JSON.

It's something we should do, but we have to be aware the document type is meant to be agnostic (the service could start returning CBOR tomorrow for example). See this comment from Norm: #3915 (comment)

It'd probably make more sense to include this functionality in Core, but now I'm even wondering if it's better to do that first (and separately) from this PR.

[GarrettBeatty]

why are we checking bedrockruntimeexception here but down below i see we are throwing InvalidOperationException when it fails?

[peterrsongg]

My long comment on testing different json responses returned by the service might not make sense depending on what this feature is supposed to do. So if this option is set, that tells Bedrock to return the response in a certain way?

I still think we shouldn't have a MockBedrockRuntime that implements IAmazonBedrockRuntime. We will have to update this class every time a new operation is released.

[peterrsongg]

So anytime BedrockRuntime adds a new operation, we have to remember to go here and update this mock class so that the operation throws NotImplementedException right? I think we should go with a different approach to mocking the service, because if that's the case then this isn't very scalable.

Also the types of responses you can test here are severely limited. The ConverseAsync just returns a default response, so these test cases are only testing the happy path. My suggestion is to mock the httpLayer so that you can test out edge cases and different responses returned by Converse

[peterrsongg]

Here is my suggestion (sorry it is a lot). This is what we did for fuzz testing

1. Create a pipeline customizer

namespace NetSdkMock
{
    public class PipelineCustomizer : IRuntimePipelineCustomizer
    {
        public string UniqueName => "MockPipeline";

        public void Customize(Type type, RuntimePipeline pipeline)
        {
            pipeline.ReplaceHandler<HttpHandler<System.Net.Http.HttpContent>>(new HttpHandler<HttpContent>(new MockHttpRequestMessageFactory(), new object()));
        }
    }

    public class MockHttpRequestMessageFactory : IHttpRequestFactory<HttpContent>
    {
        public IHttpRequest<HttpContent> CreateHttpRequest(Uri requestUri)
        {
            return new MockHttpRequest(new HttpClient(), requestUri, null);
        }

        public void Dispose()
        {
            throw new NotImplementedException();
        }
    }

    public class MockHttpRequest : HttpWebRequestMessage, IHttpRequest<HttpContent>
    {
        private IWebResponseData _webResponseData;

        public MockHttpRequest(HttpClient httpClient, Uri requestUri, IClientConfig config) : base(httpClient, requestUri, config)
        {
        }
        public new IWebResponseData GetResponse()
        {
            return this.GetResponseAsync(CancellationToken.None).Result;
        }

        public new void ConfigureRequest(IRequestContext requestContext)
        {
            _webResponseData = (IWebResponseData)((IAmazonWebServiceRequest)requestContext.OriginalRequest).RequestState["response"];
        }

        public new Task<IWebResponseData> GetResponseAsync(CancellationToken cancellationToken)
        {
            return Task.FromResult(_webResponseData);
        }
    }
}

2. Stub the web response data so that you can control the type of responses that you get

using Amazon.Runtime.Internal.Transform;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading.Tasks;

namespace NetSdkMock
{
    public class StubWebResponseData : IWebResponseData
    {
        public StubWebResponseData(string jsonResponse, Dictionary<string, string> headers)
        {
            this.StatusCode = HttpStatusCode.OK;
            this.IsSuccessStatusCode = true;
            JsonResponse = jsonResponse;
            this.Headers = headers;

            _httpResponseBody = new HttpResponseBody(jsonResponse);
        }
        public Dictionary<string, string> Headers { get; set; }

        public string JsonResponse { get; }
        private IHttpResponseBody _httpResponseBody;
        public long ContentLength { get; set; }

        public string ContentType { get; set; }

        public HttpStatusCode StatusCode { get; set; }

        public bool IsSuccessStatusCode { get; set; }

        public IHttpResponseBody ResponseBody
        {
            get
            {
                return _httpResponseBody;
            }
        }

        public string[] GetHeaderNames()
        {
            return Headers.Keys.ToArray();
        }

        public bool IsHeaderPresent(string headerName)
        {
            return this.Headers.ContainsKey(headerName);
        }

        public string GetHeaderValue(string headerName)
        {
            if (this.Headers.ContainsKey(headerName))
                return this.Headers[headerName];
            else
                return null;
        }
    }
    public class HttpResponseBody : IHttpResponseBody
    {
        private readonly string _jsonResponse;
        private Stream stream;
        public HttpResponseBody(string jsonResponse)
        {
            _jsonResponse = jsonResponse;
        }

        public void Dispose()
        {
            stream.Dispose();
        }

        public Stream OpenResponse()
        {
            stream = new MemoryStream(UTF8Encoding.UTF8.GetBytes(_jsonResponse));
            return stream;
        }

        public Task<Stream> OpenResponseAsync()
        {
            return Task.FromResult(OpenResponse());
        }
    }
}

Now in your test you can call it as normal and pass in different types of responses:

using Amazon.BedrockRuntime;
using Amazon.BedrockRuntime.Model;
using Amazon.Runtime;
using Amazon.Runtime.Internal;
using Microsoft.Extensions.AI;
using Moq;
using System.Text.Json;
namespace NetSdkMock
{
    public class MockTests : IClassFixture<ClientFixture>
    {
        private readonly ClientFixture _fixture;
        public MockTests()
        {
            _fixture = new ClientFixture();
        }
        [Fact]
        public async Task Test1()
        {
            var messages = new[] { new ChatMessage(ChatRole.User, "Test") };
            // try to test different schemas too. 
            var schemaJson = """
            {
                "type": "object",
                "properties": {
                    "name": { "type": "string" },
                    "age": { "type": "number" }
                },
                "required": ["name"]
            }
            """;
            var schemaElement = JsonDocument.Parse(schemaJson).RootElement;
            var options = new ChatOptions
            {
                ResponseFormat = ChatResponseFormat.ForJsonSchema(schemaElement,
                    schemaName: "PersonSchema",
                    schemaDescription: "A person object")
            };
            var chatClient = _fixture.BedrockRuntimeClient.AsIChatClient("claude-3");
            ConverseRequest request = new ConverseRequest();
            var interfaceType = typeof(IAmazonWebServiceRequest);
            var requestStatePropertyInfo = interfaceType.GetProperty("RequestState");
            var requestState = (Dictionary<string, object>)requestStatePropertyInfo.GetValue(request);
            //var schemaElement = JsonDocument.Parse(schemaJson).RootElement;
            //  now you can test out all different types of json responses
            var jsonResponse = """
            {
                "name":"Bob",
                "age" : 15
            }
            """;

            ChatOptions options = new ChatOptions();
            options.RawRepresentationFactory = chatClient => request;

            var webResponseData = new StubWebResponseData(jsonResponse, new Dictionary<string, string>());
            // this is where we are injecting the stubbed web response data. 
            requestState["response"] = webResponseData;
            var response = await chatClient.GetResponseAsync(messages, options).ConfigureAwait(false);

        }
    }
    public class ClientFixture: IDisposable
    {
        public ClientFixture()
        {
            RuntimePipelineCustomizerRegistry.Instance.Register(new PipelineCustomizer());
            BedrockRuntimeClient = new AmazonBedrockRuntimeClient();
        }
        public IAmazonBedrockRuntime BedrockRuntimeClient { get; private set; }

        public void Dispose()
        {
            // Cleanup after all tests in this class
            BedrockRuntimeClient.Dispose();
        }
    }
}

I spent a few hours creating a test project to make sure this code will work and it does, so I think you can use this to create many more test cases and not just test the happy path.

[peterrsongg]

Agree with garret, it should live in a utils class at the least

[peterrsongg]

is checking the error message the only way to achieve this? error messages aren't gauranteed to stay the same.

[peterrsongg]

i'm a bit confused, and maybe it is because i don't understand who is supposed to return the response in the provided schema (bedrock or us), but this test case just seems to be asserting that the tool has the correct schema set on it. Is there no way to test the actual functionality?