Automatically create nested BenchmarkGroup on access

[MilesCranmer]

Fixes #308

This lets you perform, e.g.,

suite = BenchmarkGroup()

for T in [Float32, Float64], n in [10, 100], m in [5, 20]
    suite["sum"][T][n][m] = @benchmarkable sum(x) setup=(x=randn($T, $n, $m))
end

without needing to manually create each benchmark group. Normally you would have to do something like:

suite = BenchmarkGroup()

for T in [Float32, Float64], n in [10, 100], m in [5, 20]
    !haskey(suite, "sum") && (suite["sum"] = BenchmarkGroup())
    !haskey(suite["sum"], T) && (suite["sum"][T] = BenchmarkGroup())
    !haskey(suite["sum"][T], n) && (suite["sum"][T][n] = BenchmarkGroup())

    suite["sum"][T][n][m] = @benchmarkable sum(x) setup=(x=randn($T, $n, $m))
end

which seems tedious.

This is similar to how EasyConfig.jl works, in that getindex will automatically create each required BenchmarkGroup().

The downside is that accessing the wrong key will create empty groups. Thus, there is the clear_empty! command to clean these up (I added docs too).

[MilesCranmer]

@vchuravy @ranocha would one of you be up for reviewing this? Note that this is backwards compatible so won’t cause issues in existing code. It simply makes it significantly easier to set up hierarchical benchmark suites over different param combinations.

[ranocha]

I'm not a developer/maintainer of BenchmarkTools.jl and can't really judge this. All I can say is that you should also add some tests.

[MilesCranmer]

Thanks. Will definitely add tests; just wanted to have this PR “fail fast” if this feature is a nonstarter

[ranocha]

if this feature is a nonstarter

I'm not a developer/maintainer, so I can't comment on this

[MilesCranmer]

@vchuravy would you be able to take a look at this? I can add tests if this looks okay as a feature.

[MilesCranmer]

@ararslan would you be up for taking a look at this?

[gdalle]

@vchuravy just made me a maintainer (hurray 🤣) and it turns out I ran into this very problem last week, so I'm gonna say the feature is a welcome one. Please add tests and I'll review!

[MilesCranmer]

Done!

[gdalle]

I need to fix CI cause #305 broke it for unsupported Julia versions, then we'll re-run the tests

[gdalle]

OK tests are failing outside of 1.5 and 1.7, that's not on me :)

[MilesCranmer]

Oops, fixed!

[gdalle]

@MilesCranmer do you think this is a breaking change or a new feature?

[MilesCranmer]

I think it’s a breaking change, so worth bumping the version number

[gdalle]

OK then I'll wait until we fix a few longstanding bugs

[MilesCranmer]

@gdalle I noticed a new version has not been released yet and I was wondering if you might be able to do so? Would be great to use this feature!

[gdalle]

I have been hesitant cause it's a central package and breaking changes have been made, but a recent discussion on slack convinced me to move forward. Waiting on one PR then I'll do a bit of cleanup and tag 2.0