Caching doesn't respect temperature sampling

[Jannoshh]

Problem

The current caching implementation causes issues when running propensity evaluations that rely on temperature sampling. After, e.g. running 30 samples (same input prompt) that each have a different response because of temperature sampling, only one of the responses is cached (the last I think). This means that when you rerun the evaluation, only one cached response is loaded instead of the intended diverse set of sampled outputs, which messes up the scoring completely.

This is a very annoying barrier for propensity evaluations since they typically sample many outputs for the same input to measure propensity.

[jjallaire]

The cache does use the generate config as part of it cache key so it shouldn't have crosstalk like this. Are you varying the temperature using the temperature configuration field or are you doing something more dynamic. If you could provide a simple reproducible example of code where this happens that would be super helpful in understanding the nature of the problem.

[Jannoshh]

I've been running various tests on what exactly causes the issue, I only start having the issue when I use the start using the basic_agent. Here's a report:

Reproduction

minimal_cache_bug.py:

"""
Minimal example to reproduce caching bug with temperature sampling.

Issue: When using cache with temperature > 0 and epochs > 1, all epochs
return identical responses on subsequent runs, breaking propensity evaluations.

To reproduce:
    python minimal_cache_bug.py  # First run: diverse responses
    python minimal_cache_bug.py  # Second run: all identical (BUG!)
"""

from inspect_ai import Task, task, eval as inspect_eval
from inspect_ai.dataset import Sample
from inspect_ai.model import CachePolicy, ChatMessageUser
from inspect_ai.solver import basic_agent, system_message


@task
def minimal_cache_bug() -> Task:
    """Minimal task demonstrating the caching bug with basic_agent()."""

    dataset = [
        Sample(
            input=[
                ChatMessageUser(content="Think of a random color and tell me what it is.")
            ]
        )
    ]

    cache = CachePolicy(expiry="1M")

    solver = [
        system_message("You are a helpful assistant."),
        basic_agent(cache=cache),
    ]

    return Task(
        dataset=dataset,
        solver=solver,
    )


if __name__ == "__main__":
    print("=" * 70)
    print("Minimal reproduction of caching bug with temperature sampling")
    print("Execute this script TWICE to see the bug")
    print("=" * 70)

    task = minimal_cache_bug()

    results = inspect_eval(
        tasks=[task],
        model="openai/gpt-4o-mini",
        epochs=3,
        temperature=1.0,
        log_dir="logs/cache_bug_test",
    )

    responses = []
    for log in results:
        for sample in log.samples:
            if sample.output and sample.output.completion:
                responses.append(sample.output.completion.strip())

    print(f"\nResponses:")
    for i, resp in enumerate(responses):
        print(f"  Epoch {i+1}: {resp}")
    print(f"\nUnique responses: {len(set(responses))}/3")

    print("\n" + "=" * 70)
    if len(set(responses)) == 1:
        print("❌ BUG CONFIRMED: All responses are identical (cached)!")
        print("This breaks propensity evaluations that need diverse samples.")
    elif len(set(responses)) >= 2:
        print("✓ Got diverse responses (expected on first run)")
        print("Run this script again to see the bug.")
    else:
        print("⚠️ No responses captured")
    print("=" * 70)

Run twice:

python minimal_cache_bug.py  # First run
python minimal_cache_bug.py  # Second run

First run:

Epoch 1: turquoise
Epoch 2: Turquoise
Epoch 3: turquoise
Unique: 2/3 ✓

Second run:

Epoch 1: turquoise
Epoch 2: turquoise
Epoch 3: turquoise
Unique: 1/3 ❌

[jjallaire]

Thank you for the repro -- this a great bug to have found! (it would happen anywhere you have a multi-epoch task and call get_model() rather than relying on the generate passed to solvers, which is basically any custom agent.

Fix is here: #2593

[Jannoshh]

I don't think this fixed it, I installed the current github version of inspect, and I get the same issue:

Responses:
Epoch 1: puce
Epoch 2: turquoise
Epoch 3: The random color I thought of is "chartreuse".

Unique responses: 3/3

======================================================================
✓ Got diverse responses (expected on first run)
Run this script again to see the bug.

❯ uv run python raw/minimal_cache_bug.py

Minimal reproduction of caching bug with temperature sampling
Execute this script TWICE to see the bug

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│minimal_cache_bug (1 x 3 samples): grok/grok-code-fast-1 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
temperature: 1.0, epochs: 3, dataset: (samples)

total time: 0:00:00

Log: logs/cache_bug_test/2025-10-12T19-13-37+01-00_minimal-cache-bug_S9mUzWiNhxrjUXNFwDN4rA.eval

Responses:
Epoch 1: The random color I thought of is "chartreuse".
Epoch 2: The random color I thought of is "chartreuse".
Epoch 3: The random color I thought of is "chartreuse".

Unique responses: 1/3

======================================================================
❌ BUG CONFIRMED: All responses are identical (cached)!
This breaks propensity evaluations that need diverse samples.

[Jannoshh]

@jjallaire pinging you in case this was missed.