feat: Support AWS Bedrock custom inference profiles #8801
+935
−164
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add detection and mapping for custom inference profile ARNs - Fix token limit validation for custom inference profiles (4096 instead of 8192) - Fix provider detection to use endpoint name instead of model name - Fix thinking configuration to not auto-enable for custom profiles - Add environment variable support for ARN-to-model mapping - Add comprehensive documentation and examples - Fix recursion issues in token detection functions - Add missing exports and endpoint mappings - Resolve 'Config not found' and 'Invalid URL' errors - Resolve 'thinking: Extra inputs are not permitted' error Closes danny-avila#6710
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
ESLint found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
|
Thanks for this PR! Can you resolve the ESLint issues? Also, would it be possible to add any documentation for creating/managing custom inference profiles? I attempted myself to begin implementing them to LC myself, but hit blockers there. This would help me test your implementation in order to merge. |
|
also the tests you added in |
added 3 commits
August 7, 2025 08:51
- Fix getModelMaxTokens to return undefined instead of 4096 for invalid inputs - Export BEDROCK_INFERENCE_PROFILE_MAPPINGS for test access - Update test imports to include missing functions and variables - Fix test expectations to match actual function behavior - Add comprehensive documentation for AWS Bedrock custom inference profiles - Include Python script example for creating inference profiles - Add troubleshooting guide and best practices - Document environment variable configuration - Add usage examples and API integration guide All tests now passing (80/80) Resolves ESLint issues related to our changes
- Fix tag format from 'Key=Project,Value=LibreChat' to 'key=Project,value=LibreChat' - Update documentation with correct AWS CLI syntax - Clean up Python script and fix model ARN format - Add example for multiple tags in single --tags parameter - Test and verify the corrected command works correctly The AWS CLI command now works as documented.
…iles - Remove Method 3 (AWS Console) as custom inference profiles cannot be created from console - Add note clarifying that profiles can only be created via API calls - Update section to reflect only 2 valid methods: AWS CLI and Python SDK - Fix documentation accuracy for AWS Bedrock custom inference profiles
|
thanks for raising this PR. would love to see this in main 👍 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🎉 Support AWS Bedrock Custom Inference Profiles
Problem
AWS Bedrock custom inference profiles have ARNs that don't contain model name information, causing LibreChat to fail to recognize their capabilities. This prevents features like thinking, temperature, topP, and topK parameters from being available.
Solution
Key Features
Configuration
Users can now configure custom inference profiles using the
BEDROCK_INFERENCE_PROFILE_MAPPINGSenvironment variable:Issues Resolved
Testing
All functionality has been thoroughly tested and verified to work correctly with custom inference profile ARNs.
Closes #6710