is @benchmarkset usable ? #343
Description
@benchmarkset is nowhere in the documentation and can only be found in the the API Reference.
Is there a reason why ? Like is it obselete or maybe "too new" ?
Anyway, I think it's also malfunctioning.
E.g., I would expect the refdocs example to produce 5 cases but it only produces 1:
julia> @benchmarkset "suite" for k in 1:5
@case "case $k" rand($k, $k)
end
1-element BenchmarkTools.BenchmarkGroup:
tags: []
"suite" => 1-element BenchmarkTools.BenchmarkGroup:
tags: []
"case 5" => Benchmark(evals=1, seconds=5.0, samples=10000)I would expect it to be similar with the following code block:
julia> bg = BenchmarkGroup([], "suite" => BenchmarkGroup([]));
julia> foreach(1:5) do k
bg["suite"]["case $k"] = @benchmarkable rand($k, $k)
end
julia> bg
1-element BenchmarkTools.BenchmarkGroup:
tags: []
"suite" => 5-element BenchmarkTools.BenchmarkGroup:
tags: []
"case 2" => Benchmark(evals=1, seconds=5.0, samples=10000)
"case 4" => Benchmark(evals=1, seconds=5.0, samples=10000)
"case 5" => Benchmark(evals=1, seconds=5.0, samples=10000)
"case 3" => Benchmark(evals=1, seconds=5.0, samples=10000)
"case 1" => Benchmark(evals=1, seconds=5.0, samples=10000)System
Julia Version 1.9.4
Commit 8e5136fa297 (2023-11-14 08:46 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 12 × 12th Gen Intel(R) Core(TM) i5-1235U
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-14.0.6 (ORCJIT, alderlake)
Threads: 1 on 12 virtual cores
Environment:
JULIA_IMAGE_THREADS = 1
[32113eaa] PkgBenchmark v0.2.12
similar discussion in https://discourse.julialang.org/t/use-tags-setup-teardown-with-the-benchmarkset-macro/98936/2