Skip to content

Conversation

christian-bromann
Copy link
Member

Prompt caching is only possible for 5 messages at the same time. The current version of the middleware updates the message object in directly rather than creating a new object. This causes messages to stay cached over multiple hook iterations and eventually failing due to the limit of 5.

Copy link

changeset-bot bot commented Oct 1, 2025

⚠️ No Changeset found

Latest commit: 463650d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link

vercel bot commented Oct 1, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Comments Updated (UTC)
langchainjs-docs Ready Ready Preview Comment Oct 1, 2025 4:03am
1 Skipped Deployment
Project Deployment Preview Comments Updated (UTC)
langchainjs-api-refs Ignored Ignored Oct 1, 2025 4:03am

Copy link
Member

@hntrl hntrl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know if the cache_control breakpoint only caches the content contained within a single message? Anthropic docs make it a little ambiguous..

The system automatically checks for cache hits at all previous content block boundaries

If we're only adding the cache_control param to the last message, and given that cache breakpoints are contained within a single message, then we aren't caching much of anything

@christian-bromann
Copy link
Member Author

Do we know if the cache_control breakpoint only caches the content contained within a single message?

As far as I understood it caches the message flagged with the cache property and all previous ones.

then we aren't caching much of anything

In Python we do the same here. The integration test also checks that we actually see cached tokens.

@christian-bromann christian-bromann merged commit d9ffdd7 into v1 Oct 2, 2025
34 checks passed
@christian-bromann christian-bromann deleted the cb/prompt-caching-fix branch October 2, 2025 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants