Tool's JSON Output Does Not Adhere to Provided JSON Schema

[paulhindemith]

Overview

The JSON output generated by the tool sometimes deviates from the specified input_schema.

Detailed Problem Description

The input_schema defines the expected structure for the JSON output as follows:

  {
  "additionalProperties": false,
  "properties": {
    "data": {
      "additionalProperties": false,
      "properties": {
        "format": {
          "additionalProperties": false,
          "properties": {
            "property": {
              "description": "...",
              "type": "string"
            },
            "type": {
              "enum": [
                "json"
              ],
              "description": "...",
              "type": "string"
            }
          },
          "required": [
            "type",
            "property"
          ],
          "description": "...",
          "type": "object"
        },
        "url": {
          "description": "...",
          "type": "string"
        }
      },
      "required": [
        "url",
        "format"
      ],
      "description": "...",
      "type": "object"
    },
    "...": "..."
  }
}

Based on this schema, the expected output should be in this format:

{
  "data": {
    "url": "./example.json",
    "format": {
      "property": "xxx.yyy.zzz",
      "type": "json"
    }
  },
  "...": "..."
}

However, the tool occasionally produces output that violates this schema, such as:

{
  "data": {
    "values": [
      {"a": "A", "b": 28}, {"a": "B", "b": 55}, {"a": "C", "b": 43},
      {"a": "D", "b": 91}, {"a": "E", "b": 81}, {"a": "F", "b": 53},
      {"a": "G", "b": 19}, {"a": "H", "b": 87}, {"a": "I", "b": 52}
    ]
  },
  "...": "..."
}

This output is inconsistent with the provided JSON Schema because the data object contains a values property instead of the required url and format properties.

Question

Does the tool strictly validate its JSON output against the provided JSON Schema? If so, this behavior suggests a potential bug.

Environment:

This tool is accessed from Bedrock and utilizes the Converse API for its streaming output, using boto3.

[paulhindemith]

I tried it with Boto3, but the same issue seems to be occurring with the anthropic-sdk-python as well.