Inconsistent failure to use thinking with Claude 4 Sonnet

[francisjervis]

I am getting API responses which do not contain a thinking block (and where, based on output token count, extended thinking has definitely not been applied) on some calls. The behavior is consistent across calls with the same system prompt/messages. There is no API error and the messages parameter is well formed (a prior assistant message with thinking and tool use block, user message with tool result). See console logs (starting from relevant portion of the params passed to the API):

'thinking': {'type': 'enabled', 'budget_tokens': 1024}}
01:21:51.371 Message with 'claude-sonnet-4-0' [LLM]
2025-05-26 22:21:53 - HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
Processing event in _process_evaluation_cycle: RawMessageStartEvent(message=Message(id='msg_0129AsRpyXyD6dw3EX2zxZgv', content=[], model='claude-sonnet-4-20250514', role='assistant', stop_reason=None, stop_sequence=None, type='message', usage=Usage(cache_creation_input_tokens=0, cache_read_input_tokens=0, input_tokens=3327, output_tokens=41, server_tool_use=None, service_tier='standard')), type='message_start')
Processing event in _process_evaluation_cycle: RawContentBlockStartEvent(content_block=ToolUseBlock(id='toolu_017roy3ok9vf9RsbkbXZbm6Q', input={}, name='probing_question', type='tool_use'), index=0, type='content_block_start')
Handling block start event: RawContentBlockStartEvent(content_block=ToolUseBlock(id='toolu_017roy3ok9vf9RsbkbXZbm6Q', input={}, name='probing_question', type='tool_use'), index=0, type='content_block_start')

The API call comes from the same function as one which successfully returns a thinking block. This is not an issue with how I am handling streaming events - the same stream handling code logs this earlier in the conversation:

'thinking': {'type': 'enabled', 'budget_tokens': 1024}}
01:21:34.297 Message with 'claude-sonnet-4-0' [LLM]
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1748308894.318467 18618069 fork_posix.cc:75] Other threads are currently calling into gRPC, skipping fork() handlers
I0000 00:00:1748308894.351447 18618069 fork_posix.cc:75] Other threads are currently calling into gRPC, skipping fork() handlers
2025-05-26 22:21:36 - HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
Processing event in _process_evaluation_cycle: RawMessageStartEvent(message=Message(id='msg_0131jdGMgkn2suhC33vn5jyu', content=[], model='claude-sonnet-4-20250514', role='assistant', stop_reason=None, stop_sequence=None, type='message', usage=Usage(cache_creation_input_tokens=0, cache_read_input_tokens=0, input_tokens=2736, output_tokens=2, server_tool_use=None, service_tier='standard')), type='message_start')
Processing event in _process_evaluation_cycle: RawContentBlockStartEvent(content_block=ThinkingBlock(signature='', thinking='', type='thinking'), index=0, type='content_block_start')

This is extremely undesirable due to the requirement that thinking blocks are included in the conversation history when extended thinking is enabled.

This appears to be a model-side issue based on the fact that 3.7 Sonnet does not return a tool use block sans thinking - it returns three (3) tokens, effectively an empty response:

01:54:59.528 Message with 'claude-3-7-sonnet-latest' [LLM]
2025-05-26 22:55:04 - HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
Processing event in _process_evaluation_cycle: RawMessageStartEvent(message=Message(id='msg_0144j8sbKgpsG7y4CAi1Mocj', content=[], model='claude-3-7-sonnet-20250219', role='assistant', stop_reason=None, stop_sequence=None, type='message', usage=Usage(cache_creation_input_tokens=0, cache_read_input_tokens=0, input_tokens=3155, output_tokens=3, server_tool_use=None, service_tier='standard')), type='message_start')
Processing event in _process_evaluation_cycle: RawMessageDeltaEvent(delta=Delta(stop_reason='end_turn', stop_sequence=None), type='message_delta', usage=MessageDeltaUsage(cache_creation_input_tokens=None, cache_read_input_tokens=None, input_tokens=None, output_tokens=3, server_tool_use=None))
Processing event in _process_evaluation_cycle: MessageStopEvent(type='message_stop', message=Message(id='msg_0144j8sbKgpsG7y4CAi1Mocj', content=[], model='claude-3-7-sonnet-20250219', role='assistant', stop_reason='end_turn', stop_sequence=None, type='message', usage=Usage(cache_creation_input_tokens=0, cache_read_input_tokens=0, input_tokens=3155, output_tokens=3, server_tool_use=None, service_tier='standard')))
01:55:04.595 streaming response from 'claude-3-7-sonnet-latest' took 0.03s [LLM]

Reporting it here as it's happening in Python. API results for calls with thinking enabled which do not contain a thinking block are malformed, as far as I'm concerned, and it is not mentioned in the docs that they may or may not be generated - indeed as noted above extended thinking basically does not work if they are not preserved and sent with subsequent calls. Adding code to fall back to no thinking if the model simply does not return a thinking block is not an acceptable solution.

Edit to add that I am also getting failed empty (but apparently billed) requests with other prompts, with Claude 4 Sonnet, eg:

...thinking': {'type': 'enabled', 'budget_tokens': 1024}}
04:06:18.383 Message with 'claude-sonnet-4-0' [LLM]
2025-05-27 01:06:20 - HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
04:06:20.653 streaming response from 'claude-sonnet-4-0' took 0.00s [LLM]

This is obviously not a desirable failure mode - the code has retry logic but every request fails (in very rapid succession, which is problematic given I am presumably being billed for input tokens!), if this is due to malformed inputs the API should raise an error.

[chaselambda]

Thanks for writing the issue, there's a tricky edge case where a tool result sent to the model without anything after it can, in rare cases, cause the model to output nothing. This is a known issue we're working on resolving. To confirm this is the issue you're seeing, can you add a text block on the last user turn (where it's sending a tool result) saying something like "This is the tool result"? I would expect this to resolve your issue if this is the same bug.

An even better solution - are you able to share a reproduction of the issue with the messages your sending? You can email me it - chasel+github@anthropic.com

This is news to me that this could also cause an empty thinking response.

[francisjervis]

@chaselambda here is the helper function I am using, which is designed to insert a user message with a "Creating response for {tool}" string. Just to clarify - are you saying I should add another user message after that?

    def _prepare_messages(self, state):
        """
        Prepare message history for LLM call with proper block handling.
        Ensures that each assistant/tool message with a tool_use_block is
        immediately followed by a single user message containing a tool_result.
        """
        messages = []
        processed_tools = set()
        skip_user_indices = set()
        
        # Build the filtered list
        for idx, m in enumerate(state.messages):
            if m.question_number != state.current_question:
                continue

            # Skip user messages that were already consumed as tool_result
            if idx in skip_user_indices:
                continue

            # Tool usage block handling
            if (m.type in {MessageType.ai, MessageType.tool}) and m.tool_use_block:
                tool = m.tool_use_block
                if tool.id in processed_tools:
                    continue
                processed_tools.add(tool.id)

                # Assistant message with thinking + tool_use
                content_blocks = []
                if m.thinking_block:
                    content_blocks.append(m.thinking_block)
                content_blocks.append(tool)
                messages.append({"role": "assistant", "content": content_blocks})

                # Find and consume the next user response for this tool
                user_result = None
                for j in range(idx + 1, len(state.messages)):
                    follow = state.messages[j]
                    if follow.question_number != state.current_question:
                        continue
                    if follow.type == MessageType.user:
                        user_result = follow.content
                        skip_user_indices.add(j)
                        break

                # Append tool_result wrapper
                result_content = user_result or f"Creating response for tool {tool.name}"
                messages.append({
                    "role": "user",
                    "content": [{
                        "type": "tool_result",
                        "tool_use_id": tool.id,
                        "content": result_content
                    }]
                })

            # Plain assistant messages
            elif m.type == MessageType.ai and not m.tool_use_block:
                messages.append({"role": "assistant", "content": m.content})

            # Plain tool messages
            elif m.type == MessageType.tool and not m.tool_use_block:
                messages.append({"role": "assistant", "content": m.content})

            # Remaining user messages
            elif m.type == MessageType.user:
                messages.append({"role": "user", "content": m.content})

        return messages

[chaselambda]

Yes either a new user turn or another content block to that last user turn saying something like "This is the tool result, please continue"

[francisjervis]

@chaselambda let me explain what's going on here (or at least a mildly simplified version):

• Agent is an interviewer bot
• Conversation starts with an assistant message asking a question. Assume this string is hard-coded. This is saved to Chainlit user session/state as a "vanilla" assistant message
• User answers, saved to state as user message
• LLM evaluates the answer and selects a tool - it's not always really tool use as such though, some "tools" simply trigger streaming of a response to the UI, assistant message is saved to state with tool use and thinking block (if present)
• User responds again, saved to state.
• For the next conversation turn, the last user message is treated as a "tool use result" with the id from the block saved in the previous step.

What I also don't get is that I was seeing errors with the "last assistant block must contain thinking block" message before updating the helper function above to work in that way, if that was enforced consistently requests with thinking enabled but with the last assistant message only containing a tool use block would still fail, but instead they execute with no thinking. Was there a change to make the API less opinionated or...??