Call Calude Sonnet 3.5 API with attachments in Bedrock

[rkc0818]

Discussed in #4408

^{Originally posted by rkc0818 January 28, 2025}
Hello I am trying to send two attachments to Calude 3.5 sonnet (modelId: nthropic.claude-3-5-sonnet-20240620-v1:0), I cant seem to make this work. I am sending them as follows.

payload = {
"modelId": "anthropic.claude-3-5-sonnet-20240620-v1:0",
"contentType": "application/json",
"accept": "application/json",
"body": json.dumps({
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 10000,
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Explain the diagram from the PDF and sumamrize content from the DOCX attachments"
},
*attachments
]
}
]
})
}
my attachments are created as specified below.

attachments = []

    # Add PDF file
    with open(pdf_file, 'rb') as file:
        mime_type, _ = mimetypes.guess_type(pdf_file)
        attachments.append({
            "type": "image",
            "image": {
                "source": {
                    "type": "base64",
                    "media_type": mime_type or "application/pdf",
                    "data": base64.b64encode(file.read()).decode('utf-8')
                }
            }
        })

    # Add DOCX file
    with open(docx_file, 'rb') as file:
        mime_type, _ = mimetypes.guess_type(docx_file)
        attachments.append({
            "type": "image",
            "image": {
                "source": {
                "type": "base64",
                "media_type": mime_type or "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
                "data": base64.b64encode(file.read()).decode('utf-8')
                }
            } 
        })

the error is raise error_class(parsed_response, operation_name)
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: messages.0.content.1.image.source: Field required

my invocation is
response = self.bedrock_client.invoke_model(modelId=payload["modelId"], body=payload["body"])

[khushail]

Hi @rkc0818 , thanks for reaching out. AFAIK, Anthropic's Claude sonnet model 3.5 API accepts only images and text as input, not the pdf/doc. Please refer to this AWS documentation -https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages.html#claude-messages-supported-models

Also I tried to confirm this input in AWS Console in Bedrock playground and that also accepts text as input, or image as attchments. PDF/Docs are not accepted.

let me know if you need more clarification. Thanks

[rkc0818]

Thanks for the reply. I got this working, Chat/Text Playground uses the "Converse"/"ConverseStream" APIs, these APIs readily support file attachments as is. Using invoke_model API I was able to get this working by converting PDF to Image and docx to 'text' as shown below, my docx file has tables so I had to handle tables too. The same way I can also processes xlsx files etc. Overall, I think I built a solution that can handle different file formats.

messages_content.append({
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": "image/png",
                            "data": encoded_image
                        }
                    })
                elif file_path.endswith('.docx'):
                    doc = Document(file_path)
                    text_content = []
                    
                    for paragraph in doc.paragraphs:
                        text_content.append(paragraph.text)
                    
                    # Add text content
                    if text_content:
                        messages_content.append({
                            "type": "text",
                            "text": "\n".join(text_content)
                        })
                    
                    # Handle tables if present
                    if doc.tables:
                        table_text = []
                        for i, table in enumerate(doc.tables, 1):
                            table_text.append(f"\nTable {i}:")
                            for row in table.rows:
                                row_text = [cell.text for cell in row.cells]
                                table_text.append(" | ".join(row_text))
                        
                        if table_text:
                            messages_content.append({
                                "type": "text",
                                "text": "\n".join(table_text)
                            })

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.