anthropic pdf #1074
anthropic pdf #1074
Conversation
|
@narengogi - Can you please fix the examples added in the description. Not sure if they were updated by Matter bot or were present in the original description. |
|
here are request bodies to test {
"model": "claude-3-5-sonnet-20240620",
"max_tokens": 200,
"stream": false,
"messages": [
{
"role": "system",
"content": "Say Hi"
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "what is in this pdf"
},
{
"type": "file",
"file": {
"file_url": "https://pdfobject.com/pdf/sample.pdf",
"mime_type": "application/pdf"
}
}
]
}
]
}pdf base64 {
"model": "claude-3-5-sonnet-20240620",
"max_tokens": 200,
"stream": false,
"messages": [
{
"role": "system",
"content": "Say Hi"
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "what is in this pdf"
},
{
"type": "file",
"file": {
"mime_type": "application/pdf",
"file_data": "BASE_64_PDF"
}
}
]
}
]
}plain text document {
"model": "claude-3-5-sonnet-20240620",
"max_tokens": 200,
"stream": false,
"messages": [
{
"role": "system",
"content": "Say Hi"
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "what is in this pdf"
},
{
"type": "file",
"file": {
"mime_type": "text/plain",
"file_data": "hello how are you sir"
}
}
]
}
]
} |
|
Summary By MatterAI
🔄 What ChangedAdded support for PDF and plain text document handling in Anthropic and Bedrock providers. The PR introduces new interfaces for PDF content items (URL and Base64) and plain text content items, along with helper functions to transform and append file content items to messages. 🔍 Impact of the ChangeEnables the system to process and send PDF documents and plain text files to AI models through both direct URLs and base64-encoded data. Different providers have varying levels of support: Vertex Anthropic doesn't support plain text and PDF URLs, while Bedrock only supports S3 URLs and doesn't support plain text. 📁 Total Files Changed3 files modified with 109 additions and 3 deletions:
🧪 Test AddedN/A - No tests were added in this PR, though sample request bodies were provided for manual testing. 🔒Security VulnerabilitiesN/A - No security vulnerabilities identified. Type of Change
How Has This Been Tested?
Checklist
Sequence DiagramsequenceDiagram
participant Client
participant API
participant AnthropicProvider
participant BedrockProvider
Client->>API: POST /chat/completions with file content
Note over Client,API: Content can include PDF or text files
API->>AnthropicProvider: Process request with file content
Note over AnthropicProvider: transformAndAppendFileContentItem()
alt PDF URL
AnthropicProvider-->>AnthropicProvider: Create document with URL source
Note over AnthropicProvider: {type: 'document', source: {type: 'url', url: item.file.file_url}}
else PDF Base64
AnthropicProvider-->>AnthropicProvider: Create document with base64 source
Note over AnthropicProvider: {type: 'document', source: {type: 'base64', data: item.file.file_data}}
else Plain Text
AnthropicProvider-->>AnthropicProvider: Create document with text source
Note over AnthropicProvider: {type: 'document', source: {type: 'text', data: item.file.file_data}}
end
API->>BedrockProvider: Process request with file content
Note over BedrockProvider: getMessageContent()
alt PDF URL (S3)
BedrockProvider-->>BedrockProvider: Create document with s3Location
Note over BedrockProvider: {document: {format: fileFormat, source: {s3Location: {uri: item.file.file_url}}}}
else PDF Base64
BedrockProvider-->>BedrockProvider: Create document with bytes
Note over BedrockProvider: {document: {format: fileFormat, source: {bytes: item.file.file_data}}}
end
AnthropicProvider-->>API: Return transformed request for Anthropic API
BedrockProvider-->>API: Return transformed request for Bedrock API
API-->>Client: Return model response
|
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
|
Summary By MatterAI
🔄 What ChangedThis PR adds support for PDF and plain text document handling in Anthropic and Bedrock providers. It introduces new interfaces for PDF content items (URL-based and base64-encoded) and plain text content items, along with transformation functions to process these file types appropriately for each provider. 🔍 Impact of the ChangeThis enhancement allows the system to handle PDF documents and plain text files in addition to images, expanding the types of content that can be processed by the AI models. The implementation supports both URL-based files and directly encoded file data. 📁 Total Files Changed
🧪 Test AddedManual testing was performed with Anthropic, Vertex Anthropic, and Bedrock Anthropic providers using various test cases:
🔒Security VulnerabilitiesNo security vulnerabilities identified. MotivationTo enhance the system's capability to process different file types, specifically PDF documents and plain text files, across multiple AI providers. Type of Change
How Has This Been Tested?
Screenshots (if applicable)N/A Checklist
Related IssuesN/A Sequence DiagramsequenceDiagram
participant Client
participant API
participant AnthropicProvider
participant BedrockProvider
Client->>API: POST /chat/completions
Note over Client,API: Request with file content (PDF/text)
API->>AnthropicProvider: transformAndAppendFileContentItem(item, transformedMessage)
Note over AnthropicProvider: Process file content based on type
alt File URL provided
AnthropicProvider-->>AnthropicProvider: Create document with URL source
else File data provided
AnthropicProvider-->>AnthropicProvider: Determine content type (text/base64)
AnthropicProvider-->>AnthropicProvider: Create document with data source
end
API->>BedrockProvider: getMessageContent(message)
Note over BedrockProvider: Process file content for Bedrock
alt File URL provided
BedrockProvider-->>BedrockProvider: Create document with s3Location source
else File data provided
BedrockProvider-->>BedrockProvider: Create document with bytes source
end
alt Anthropic Provider Selected
API-->>AnthropicProvider: Process request with file content
AnthropicProvider-->>Client: Return response
else Bedrock Provider Selected
API-->>BedrockProvider: Process request with file content
BedrockProvider-->>Client: Return response
end
|
2 participants
Author Description
so couple of things in this PR,
I've added two new parameters in the chat completions request body file content part
I've tested with anthropic, vertex anthropic and bedrock anthropic
note: vertex anthropic does not support plain text and pdf urls, bedrock does not support plain text, and only supports s3urls
here are request bodies to test
pdf url
{ "model": "claude-3-5-sonnet-20240620", "max_tokens": 200, "stream": false, "messages": } ] }pdf base64
{ "model": "claude-3-5-sonnet-20240620", "max_tokens": 200, "stream": false, "messages": } ] }plain text document
{ "model": "claude-3-5-sonnet-20240620", "max_tokens": 200, "stream": false, "messages": } ] }Summary By MatterAI
🔄 What Changed
This PR adds support for PDF files in Anthropic and Bedrock providers by introducing two new parameters in the chat completions request body file content part:
file_urlandmime_type. The implementation supports both PDF URLs and base64-encoded PDFs, as well as plain text documents.🔍 Impact of the Change
This enhancement allows users to send PDF documents to Anthropic and Bedrock models for analysis. Different providers have varying support: Vertex Anthropic doesn't support plain text and PDF URLs, while Bedrock only supports S3 URLs for PDFs.
📁 Total Files Changed
3 files changed with 120 additions and 3 deletions:
src/providers/anthropic/chatComplete.ts: Added PDF handling for Anthropicsrc/providers/bedrock/chatComplete.ts: Added PDF handling for Bedrocksrc/types/requestBody.ts: Updated content type interfaces🧪 Test Added
Manual testing was performed with Anthropic, Vertex Anthropic, and Bedrock Anthropic providers using various request bodies (PDF URL, PDF base64, plain text).
🔒 Security Vulnerabilities
N/A
Type of Change
How Has This Been Tested?
Screenshots (if applicable)
N/A
Checklist
Related Issues
N/A
Quality Recommendations
Add error handling for unsupported file types or formats
Add validation for file_url and file_data to ensure they are properly formatted
Consider adding unit tests to verify the PDF handling functionality
Add documentation comments for the new interfaces and functions
Sequence Diagram
sequenceDiagram participant Client participant Gateway participant AnthropicProvider participant BedrockProvider Client->>Gateway: POST /chat/completions Note over Client,Gateway: Request with file_url or file_data alt Anthropic Provider Gateway->>AnthropicProvider: transformAndAppendFileContentItem() Note over AnthropicProvider: Process file content based on type alt PDF URL AnthropicProvider-->>AnthropicProvider: Create AnthropicUrlPdfContentItem Note over AnthropicProvider: { type: 'document', source: { type: 'url', url: file_url } } else PDF Base64 AnthropicProvider-->>AnthropicProvider: Create AnthropicBase64PdfContentItem Note over AnthropicProvider: { type: 'document', source: { type: 'base64', data: file_data, media_type: mime_type } } else Plain Text AnthropicProvider-->>AnthropicProvider: Create AnthropicPlainTextContentItem Note over AnthropicProvider: { type: 'document', source: { type: 'text', data: file_data, media_type: mime_type } } end else Bedrock Provider Gateway->>BedrockProvider: getMessageContent() Note over BedrockProvider: Process file content based on type alt File URL (S3) BedrockProvider-->>BedrockProvider: Create document with s3Location Note over BedrockProvider: { document: { format: fileFormat, name: UUID, source: { s3Location: { uri: file_url } } } } else File Data BedrockProvider-->>BedrockProvider: Create document with bytes Note over BedrockProvider: { document: { format: fileFormat, name: UUID, source: { bytes: file_data } } } end end Gateway-->>Client: Return completion response#1075