Repeated `@btime` with different setup behaves bad

[sgaure]

function sumN(N)
    s = zero(N)
    for i in 1:N
        s += i
    end
    s
end

julia> @btime sumN(N) setup=(N=1_000_000)
  2.424 ns (0 allocations: 0 bytes)
500000500000

julia> @btime sumN(N) setup=(N=1_000_000e0)
  2.424 ns (0 allocations: 0 bytes)
500000500000

julia> @btime sumN(N) setup=(N=1_000_001e0)
  1.330 ms (0 allocations: 0 bytes)
5.00001500001e11

julia> @btime sumN(N) setup=(N=1_000_001)
  1.330 ms (0 allocations: 0 bytes)
5.00001500001e11

julia> @btime sumN(M) setup=(M=1_000_001)
  1.993 ns (0 allocations: 0 bytes)
500001500001

The Float64 trial should run slowly, and return a Float64, whereas the Int trial should run in a couple of ns and return an Int. Is there some caching of the setup going on, which uses == instead of ===?

[vchuravy]

There is a cache here

BenchmarkTools.jl/src/execution.jl

Lines 557 to 558 in 8b64635

	samplefunc = get!(samplefunc_cache, samplefunc_key) do
	corefunc = gensym("core")

[sgaure]

Ah, that explains it, I think. samplefunc_key is a Vector containing Exprs, samplefunc_cache is a Dict, and Expr(:tuple, 1) == Expr(:tuple, 1e0). Making it an IdDict would defeat its purpose since Expr(:tuple, 1) !== Expr(:tuple, 1). Changing this behaviour requires some more coding which probably isn't worth it.

[vchuravy]

I wonder if we should make the key a tuple instead of a vector.

[vtjnash]

This is probably fixed in v1.12