The issue is with these lines here: https://github.com/JuliaCI/BenchmarkTools.jl/blob/0941fd1c458fff6db6a90d4ca098f38381bed93f/src/execution.jl#L113-L117 The problem is that we time out if the warmup takes a long time, because we start the clock before the warmup