ingest_knowledge_base_documents fails when using S3 metadata

[AymaneHan1]

I'm trying to ingest documents into a Bedrock knowledge base via URI, with each document having its own corresponding metadata file. This metadata contains fields like user ID and chat ID that I want to use for filtering during retrieval.

While ingesting documents without metadata works fine, the process fails whenever I include metadata.

import boto3
import json
import os
import uuid

# Setup
file_path = "../data/test_2.pdf"
user_id = "009"
knowledge_id = "008"
s3_bucket = "my-bucket-name"

s3_client = boto3.client("s3")

# Keep the original filename
file_name = os.path.basename(file_path)
s3_key = f"data/{file_name}"

# I've tried several metadata formats:

# Format 1:
# metadata = {
#     "metadataAttributes": {
#         "user_id": {"stringValue": user_id},
#         "knowledge_id": {"stringValue": knowledge_id},
#     }
# }

# Format 2:
# metadata = {
#     "metadataAttributes": {
#         "user_id": {"type": "STRING", "value": user_id},
#         "knowledge_id": {"type": "STRING", "value": knowledge_id},
#     }
# }

# Format 3:
metadata = {
    "metadataAttributes": {
        "user_id": user_id,
        "knowledge_id": knowledge_id,
    }
}
# Format 4:
# metadata_file = {"user_id": user_id, "knowledge_id": knowledge_id}

metadata_file = {"user_id": user_id, "knowledge_id": knowledge_id}

# Upload file to S3
print(f"Uploading {file_path} to s3://{s3_bucket}/{s3_key}")
s3_client.upload_file(
    file_path,
    s3_bucket,
    s3_key,
    ExtraArgs={"Metadata": metadata_file},
)

# Upload metadata as a separate JSON file
metadata_key = f"data/{file_name}.metadata.json"
s3_client.put_object(
    Body=json.dumps(metadata),
    Bucket=s3_bucket,
    Key=metadata_key,
    ContentType="application/json",
)

# S3 URIs for document and metadata
s3_uri = f"s3://{s3_bucket}/{s3_key}"
metadata_uri = f"s3://{s3_bucket}/{metadata_key}"

# Initialize Bedrock client
bedrock_agent_client = boto3.client('bedrock-agent')

# Generate client token
client_token = str(uuid.uuid4())

# Attempt to ingest documents with metadata
response = bedrock_agent_client.ingest_knowledge_base_documents(
    clientToken=client_token,
    knowledgeBaseId="my-knowledge-base-id",
    dataSourceId="my-data-source-id",
    documents=[
        {
            "content": {"dataSourceType": "S3", "s3": {"s3Location": {"uri": s3_uri}}},
            "metadata": {"type": "S3_LOCATION", "s3Location": {"uri": metadata_uri}},
        }
    ],
)

The ingestion works when I remove the metadata field from the request, but fails with the metadata included. I've tried multiple formats for the metadata JSON structure without success.

What is the correct format for metadata when using ingest_knowledge_base_documents?

Thank you 🙏

[jerrold110]

Same issue here using Boto3 1.40.52

Data and metadata (json) files are on S3, response does not return any errors, but on the console it shows that the ingestion for that particular file was a failure. Ingestion only works when metadata is not included.