Zygote 0.6.41 breaks the tests

[simsurace]

Tested on Julia 1.8-rc1.
Pinning Zygote to 0.6.40 passes all tests.

(ApproximateGPs) pkg> test
     Testing ApproximateGPs
┌ Warning: Could not use exact versions of packages in manifest, re-resolving
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1581
      Status `/tmp/jl_wk0PO1/Project.toml`
  [99985d1d] AbstractGPs v0.5.13
  [298c2ebc] ApproximateGPs v0.4.2 `~/.julia/dev/ApproximateGPs.jl`
  [d360d2e6] ChainRulesCore v1.15.1
  [cdddcdb0] ChainRulesTestUtils v1.8.1
  [31c24e10] Distributions v0.25.63
  [26cc04aa] FiniteDifferences v0.12.24
  [587475ba] Flux v0.13.3
  [c8e1da08] IterTools v1.4.0
  [2ab3a3ac] LogExpFunctions v0.3.15
  [429524aa] Optim v1.7.0
  [90014a1f] PDMats v0.11.14
  [e88e6eb3] Zygote v0.6.41
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [9a3f8284] Random `@stdlib/Random`
  [8dfed614] Test `@stdlib/Test`
      Status `/tmp/jl_wk0PO1/Manifest.toml`
  [621f4979] AbstractFFTs v1.1.0
  [99985d1d] AbstractGPs v0.5.13
  [7d9f7c33] Accessors v0.1.14
  [79e6a3ab] Adapt v3.3.3
  [298c2ebc] ApproximateGPs v0.4.2 `~/.julia/dev/ApproximateGPs.jl`
  [dce04be8] ArgCheck v2.3.0
  [4fba245c] ArrayInterface v6.0.19
  [30b0a656] ArrayInterfaceCore v0.1.14
  [ab4f0b2a] BFloat16s v0.2.0
  [198e06fe] BangBang v0.3.36
  [9718e550] Baselet v0.1.1
  [fa961155] CEnum v0.4.2
  [052768ef] CUDA v3.11.0
  [49dc2e85] Calculus v0.5.1
  [082447d4] ChainRules v1.36.2
  [d360d2e6] ChainRulesCore v1.15.1
  [cdddcdb0] ChainRulesTestUtils v1.8.1
  [9e997f8a] ChangesOfVariables v0.1.3
  [bbf7d656] CommonSubexpressions v0.3.0
⌅ [34da2185] Compat v3.45.0
  [a33af91c] CompositionsBase v0.1.1
  [187b0558] ConstructionBase v1.3.1
  [6add18c4] ContextVariablesX v0.1.2
  [9a962f9c] DataAPI v1.10.0
  [864edb3b] DataStructures v0.18.13
  [e2d170a0] DataValueInterfaces v1.0.0
  [244e2a9f] DefineSingletons v0.1.2
  [b429d917] DensityInterface v0.4.0
  [163ba53b] DiffResults v1.0.3
  [b552c78f] DiffRules v1.11.0
  [b4f34e82] Distances v0.10.7
  [31c24e10] Distributions v0.25.63
⌅ [ffbed154] DocStringExtensions v0.8.6
  [fa6b7ba4] DualNumbers v0.6.8
  [e2ba6199] ExprTools v0.1.8
  [cc61a311] FLoops v0.2.0
  [b9860ae5] FLoopsBase v0.1.1
  [442a2c76] FastGaussQuadrature v0.4.9
  [1a297f60] FillArrays v0.13.2
  [6a86dc24] FiniteDiff v2.13.0
  [26cc04aa] FiniteDifferences v0.12.24
  [587475ba] Flux v0.13.3
  [9c68100b] FoldsThreads v0.1.1
  [f6369f11] ForwardDiff v0.10.30
  [069b7b12] FunctionWrappers v1.1.2
⌅ [d9f16b24] Functors v0.2.8
  [6031954c] GPLikelihoods v0.4.3
  [0c68f7d7] GPUArrays v8.4.0
  [46192b85] GPUArraysCore v0.1.0
  [61eb1bfa] GPUCompiler v0.16.1
  [34004b35] HypergeometricFunctions v0.3.10
  [7869d1d1] IRTools v0.4.6
  [615f187c] IfElse v0.1.1
  [22cec73e] InitialValues v0.3.1
  [3587e190] InverseFunctions v0.1.7
  [92d709cd] IrrationalConstants v0.1.1
  [c8e1da08] IterTools v1.4.0
  [82899510] IteratorInterfaceExtensions v1.0.0
  [692b3bcd] JLLWrappers v1.4.1
  [b14d175d] JuliaVariables v0.2.4
  [ec8451be] KernelFunctions v0.10.40
  [929cbde3] LLVM v4.14.0
  [d3d80556] LineSearches v7.1.1
  [2ab3a3ac] LogExpFunctions v0.3.15
  [d8e11817] MLStyle v0.4.12
  [f1d291b0] MLUtils v0.2.9
  [1914dd2f] MacroTools v0.5.9
  [128add7d] MicroCollections v0.1.2
  [e1d29d7a] Missings v1.0.2
  [d41bc354] NLSolversBase v7.8.2
  [872c559c] NNlib v0.8.8
  [a00861dc] NNlibCUDA v0.2.3
⌅ [77ba4419] NaNMath v0.3.7
  [71a1bf82] NameResolution v0.1.5
  [429524aa] Optim v1.7.0
  [3bd65402] Optimisers v0.2.7
  [bac558e1] OrderedCollections v1.4.1
  [90014a1f] PDMats v0.11.14
  [d96e819e] Parameters v0.12.3
  [85a6dd25] PositiveFactorizations v0.2.4
  [21216c6a] Preferences v1.3.0
  [8162dcfd] PrettyPrint v0.2.0
  [33c8b6b6] ProgressLogging v0.1.4
  [1fd47b50] QuadGK v2.4.2
  [74087812] Random123 v1.5.0
  [e6cf234a] RandomNumbers v1.5.3
  [c1ae055f] RealDot v0.1.0
  [3cdcf5f2] RecipesBase v1.2.1
  [189a3867] Reexport v1.2.2
  [ae029012] Requires v1.3.0
  [708f8203] Richardson v1.4.0
  [79098fc4] Rmath v0.7.0
⌅ [efcf1570] Setfield v0.8.2
  [605ecd9f] ShowCases v0.1.0
  [a2af1166] SortingAlgorithms v1.0.1
  [276daf66] SpecialFunctions v2.1.6
  [171d559e] SplittablesBase v0.1.14
  [aedffcd0] Static v0.7.5
  [90137ffa] StaticArrays v1.5.0
  [1e83bf80] StaticArraysCore v1.0.1
  [82ae8749] StatsAPI v1.4.0
  [2913bbd2] StatsBase v0.33.18
  [4c63d2b9] StatsFuns v1.0.1
  [3783bdb8] TableTraits v1.0.1
  [bd369af6] Tables v1.7.0
  [62fd8b95] TensorCore v0.1.1
  [a759f4b9] TimerOutputs v0.5.20
  [28d57a85] Transducers v0.4.73
  [3a884ed6] UnPack v1.0.2
  [e88e6eb3] Zygote v0.6.41
  [700de1a5] ZygoteRules v0.2.2
  [dad2f222] LLVMExtra_jll v0.0.16+0
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [f50d1b31] Rmath_jll v0.3.0+0
  [0dad84c5] ArgTools v1.1.1 `@stdlib/ArgTools`
  [56f22d72] Artifacts `@stdlib/Artifacts`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [ade2ca70] Dates `@stdlib/Dates`
  [8bb1440f] DelimitedFiles `@stdlib/DelimitedFiles`
  [8ba89e20] Distributed `@stdlib/Distributed`
  [f43a241f] Downloads v1.6.0 `@stdlib/Downloads`
  [7b1f6079] FileWatching `@stdlib/FileWatching`
  [9fa8497b] Future `@stdlib/Future`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [4af54fe1] LazyArtifacts `@stdlib/LazyArtifacts`
  [b27032c2] LibCURL v0.6.3 `@stdlib/LibCURL`
  [76f85450] LibGit2 `@stdlib/LibGit2`
  [8f399da3] Libdl `@stdlib/Libdl`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [a63ad114] Mmap `@stdlib/Mmap`
  [ca575930] NetworkOptions v1.2.0 `@stdlib/NetworkOptions`
  [44cfe95a] Pkg v1.8.0 `@stdlib/Pkg`
  [de0858da] Printf `@stdlib/Printf`
  [3fa0cd96] REPL `@stdlib/REPL`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA v0.7.0 `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [1a1011a3] SharedArrays `@stdlib/SharedArrays`
  [6462fe0b] Sockets `@stdlib/Sockets`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [10745b16] Statistics `@stdlib/Statistics`
  [4607b0f0] SuiteSparse `@stdlib/SuiteSparse`
  [fa267f1f] TOML v1.0.0 `@stdlib/TOML`
  [a4e569a6] Tar v1.10.0 `@stdlib/Tar`
  [8dfed614] Test `@stdlib/Test`
  [cf7118a7] UUIDs `@stdlib/UUIDs`
  [4ec0a83e] Unicode `@stdlib/Unicode`
  [e66e0078] CompilerSupportLibraries_jll v0.5.2+0 `@stdlib/CompilerSupportLibraries_jll`
  [deac9b47] LibCURL_jll v7.81.0+0 `@stdlib/LibCURL_jll`
  [29816b5a] LibSSH2_jll v1.10.2+0 `@stdlib/LibSSH2_jll`
  [c8ffd9c3] MbedTLS_jll v2.28.0+0 `@stdlib/MbedTLS_jll`
  [14a3606d] MozillaCACerts_jll v2022.2.1 `@stdlib/MozillaCACerts_jll`
  [4536629a] OpenBLAS_jll v0.3.20+0 `@stdlib/OpenBLAS_jll`
  [05823500] OpenLibm_jll v0.8.1+0 `@stdlib/OpenLibm_jll`
  [83775a58] Zlib_jll v1.2.12+3 `@stdlib/Zlib_jll`
  [8e850b90] libblastrampoline_jll v5.1.0+0 `@stdlib/libblastrampoline_jll`
  [8e850ede] nghttp2_jll v1.41.0+1 `@stdlib/nghttp2_jll`
  [3f19e933] p7zip_jll v17.4.0+0 `@stdlib/p7zip_jll`
        Info Packages marked with ⌅ have new versions available but cannot be upgraded. To see why use `status --outdated`
Precompiling project...
  7 dependencies successfully precompiled in 36 seconds. 114 already precompiled.
     Testing Running tests...
[ Info: Packages Loaded
optimised posterior: Error During Test at /home/simone_a/.julia/dev/ApproximateGPs.jl/test/SparseVariationalApproximationModule.jl:141
  Got exception outside of a @test
  MethodError: no method matching UpperTriangular(::NamedTuple{(:data,), Tuple{UpperTriangular{Float64, Matrix{Float64}}}})
  Closest candidates are:
    UpperTriangular(::UpperTriangular) at ~/.julia/juliaup/julia-1.8.0-rc1+0~x64/share/julia/stdlib/v1.8/LinearAlgebra/src/triangular.jl:21
    UpperTriangular(::AbstractMatrix) at ~/.julia/juliaup/julia-1.8.0-rc1+0~x64/share/julia/stdlib/v1.8/LinearAlgebra/src/triangular.jl:23
    UpperTriangular(::AbstractThunk) at ~/.julia/packages/ChainRulesCore/16PWJ/src/tangent_types/thunks.jl:59
  Stacktrace:
    [1] (::Zygote.var"#719#720")(Δ::NamedTuple{(:data,), Tuple{UpperTriangular{Float64, Matrix{Float64}}}})
      @ Zygote ~/.julia/packages/Zygote/IoW2g/src/lib/array.jl:466
    [2] (::Zygote.var"#3145#back#721"{Zygote.var"#719#720"})(Δ::NamedTuple{(:data,), Tuple{UpperTriangular{Float64, Matrix{Float64}}}})
      @ Zygote ~/.julia/packages/ZygoteRules/AIbCs/src/adjoint.jl:67
    [3] Pullback
      @ ~/.julia/juliaup/julia-1.8.0-rc1+0~x64/share/julia/stdlib/v1.8/LinearAlgebra/src/cholesky.jl:190 [inlined]
    [4] (::typeof(∂(_chol!)))(Δ::Tuple{NamedTuple{(:data,), Tuple{UpperTriangular{Float64, Matrix{Float64}}}}, Nothing})
      @ Zygote ~/.julia/packages/Zygote/IoW2g/src/compiler/interface2.jl:0
    [5] Pullback
      @ ~/.julia/juliaup/julia-1.8.0-rc1+0~x64/share/julia/stdlib/v1.8/LinearAlgebra/src/cholesky.jl:269 [inlined]
    [6] (::typeof(∂(#cholesky!#154)))(Δ::NamedTuple{(:factors, :uplo, :info), Tuple{UpperTriangular{Float64, Matrix{Float64}}, Nothing, Nothing}})
      @ Zygote ~/.julia/packages/Zygote/IoW2g/src/compiler/interface2.jl:0
    [7] Pullback
      @ ~/.julia/juliaup/julia-1.8.0-rc1+0~x64/share/julia/stdlib/v1.8/LinearAlgebra/src/cholesky.jl:268 [inlined]
  --- the last 2 lines are repeated 1 more time ---
   [10] (::typeof(∂(cholesky!##kw)))(Δ::NamedTuple{(:factors, :uplo, :info), Tuple{UpperTriangular{Float64, Matrix{Float64}}, Nothing, Nothing}})
      @ Zygote ~/.julia/packages/Zygote/IoW2g/src/compiler/interface2.jl:0
   [11] Pullback
      @ ~/.julia/juliaup/julia-1.8.0-rc1+0~x64/share/julia/stdlib/v1.8/LinearAlgebra/src/cholesky.jl:402 [inlined]
   [12] (::typeof(∂(#cholesky#164)))(Δ::NamedTuple{(:factors, :uplo, :info), Tuple{UpperTriangular{Float64, Matrix{Float64}}, Nothing, Nothing}})
      @ Zygote ~/.julia/packages/Zygote/IoW2g/src/compiler/interface2.jl:0
   [13] Pullback (repeats 2 times)
      @ ~/.julia/juliaup/julia-1.8.0-rc1+0~x64/share/julia/stdlib/v1.8/LinearAlgebra/src/cholesky.jl:402 [inlined]
   [14] Pullback
      @ ~/.julia/dev/ApproximateGPs.jl/src/utils.jl:17 [inlined]
   [15] (::typeof(∂(_chol_cov)))(Δ::NamedTuple{(:factors, :uplo, :info), Tuple{UpperTriangular{Float64, Matrix{Float64}}, Nothing, Nothing}})
      @ Zygote ~/.julia/packages/Zygote/IoW2g/src/compiler/interface2.jl:0
   [16] Pullback
      @ ~/.julia/dev/ApproximateGPs.jl/src/SparseVariationalApproximationModule.jl:181 [inlined]
   [17] (::typeof(∂(posterior)))(Δ::NamedTuple{(:approx, :prior, :data), Tuple{NamedTuple{(:fz, :q), Tuple{NamedTuple{(:f, :x, :Σy), Tuple{Nothing, Vector{Float64}, Nothing}}, Nothing}}, NamedTuple{(:mean, :kernel), Tuple{Nothing, NamedTuple{(:kernel, :σ²), Tuple{NamedTuple{(:kernel, :transform), Tuple{Nothing, NamedTuple{(:s,), Tuple{Vector{Float64}}}}}, Vector{Float64}}}}}, NamedTuple{(:Kuu, :B, :α), Tuple{NamedTuple{(:factors, :uplo, :info), Tuple{UpperTriangular{Float64, Adjoint{Float64, Matrix{Float64}}}, Nothing, Nothing}}, LowerTriangular{Float64, Adjoint{Float64, Matrix{Float64}}}, Vector{Float64}}}}})
      @ Zygote ~/.julia/packages/Zygote/IoW2g/src/compiler/interface2.jl:0
   [18] Pullback
      @ ~/.julia/dev/ApproximateGPs.jl/src/SparseVariationalApproximationModule.jl:353 [inlined]
   [19] (::typeof(∂(#elbo#4)))(Δ::Float64)
      @ Zygote ~/.julia/packages/Zygote/IoW2g/src/compiler/interface2.jl:0
   [20] Pullback
      @ ~/.julia/dev/ApproximateGPs.jl/src/SparseVariationalApproximationModule.jl:340 [inlined]
   [21] (::typeof(∂(elbo##kw)))(Δ::Float64)
      @ Zygote ~/.julia/packages/Zygote/IoW2g/src/compiler/interface2.jl:0
   [22] Pullback
      @ ~/.julia/dev/ApproximateGPs.jl/src/SparseVariationalApproximationModule.jl:316 [inlined]
   [23] (::typeof(∂(#elbo#2)))(Δ::Float64)
      @ Zygote ~/.julia/packages/Zygote/IoW2g/src/compiler/interface2.jl:0
   [24] Pullback
      @ ~/.julia/dev/ApproximateGPs.jl/src/SparseVariationalApproximationModule.jl:307 [inlined]
   [25] (::typeof(∂(elbo)))(Δ::Float64)
      @ Zygote ~/.julia/packages/Zygote/IoW2g/src/compiler/interface2.jl:0
   [26] Pullback
      @ ~/.julia/dev/ApproximateGPs.jl/test/SparseVariationalApproximationModule.jl:170 [inlined]
   [27] (::typeof(∂(λ)))(Δ::Float64)
      @ Zygote ~/.julia/packages/Zygote/IoW2g/src/compiler/interface2.jl:0
   [28] Pullback
      @ ~/.julia/dev/ApproximateGPs.jl/test/SparseVariationalApproximationModule.jl:180 [inlined]
   [29] (::typeof(∂(λ)))(Δ::Float64)
      @ Zygote ~/.julia/packages/Zygote/IoW2g/src/compiler/interface2.jl:0
   [30] #216
      @ ~/.julia/packages/Zygote/IoW2g/src/lib/lib.jl:207 [inlined]
   [31] #1909#back
      @ ~/.julia/packages/ZygoteRules/AIbCs/src/adjoint.jl:67 [inlined]
   [32] Pullback
      @ ~/.julia/packages/Flux/js6mP/src/optimise/train.jl:120 [inlined]
   [33] (::typeof(∂(λ)))(Δ::Float64)
      @ Zygote ~/.julia/packages/Zygote/IoW2g/src/compiler/interface2.jl:0
   [34] (::Zygote.var"#97#98"{Params{Zygote.Buffer{Any, Vector{Any}}}, typeof(∂(λ)), Zygote.Context})(Δ::Float64)
      @ Zygote ~/.julia/packages/Zygote/IoW2g/src/compiler/interface.jl:357
   [35] gradient(f::Function, args::Params{Zygote.Buffer{Any, Vector{Any}}})
      @ Zygote ~/.julia/packages/Zygote/IoW2g/src/compiler/interface.jl:76
   [36] macro expansion
      @ ~/.julia/packages/Flux/js6mP/src/optimise/train.jl:119 [inlined]
   [37] macro expansion
      @ ~/.julia/packages/ProgressLogging/6KXlp/src/ProgressLogging.jl:328 [inlined]
   [38] train!(loss::Function, ps::Params{Zygote.Buffer{Any, Vector{Any}}}, data::IterTools.NCycle{Vector{Tuple{Vector{Float64}, Vector{Float64}}}}, opt::Flux.Optimise.ADAM; cb::Flux.Optimise.var"#38#41")
      @ Flux.Optimise ~/.julia/packages/Flux/js6mP/src/optimise/train.jl:117
   [39] train!(loss::Function, ps::Params{Zygote.Buffer{Any, Vector{Any}}}, data::IterTools.NCycle{Vector{Tuple{Vector{Float64}, Vector{Float64}}}}, opt::Flux.Optimise.ADAM)
      @ Flux.Optimise ~/.julia/packages/Flux/js6mP/src/optimise/train.jl:113
   [40] macro expansion
      @ ~/.julia/dev/ApproximateGPs.jl/test/SparseVariationalApproximationModule.jl:180 [inlined]
   [41] macro expansion
      @ ~/.julia/juliaup/julia-1.8.0-rc1+0~x64/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [42] macro expansion
      @ ~/.julia/dev/ApproximateGPs.jl/test/SparseVariationalApproximationModule.jl:142 [inlined]
   [43] macro expansion
      @ ~/.julia/juliaup/julia-1.8.0-rc1+0~x64/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [44] macro expansion
      @ ~/.julia/dev/ApproximateGPs.jl/test/SparseVariationalApproximationModule.jl:100 [inlined]
   [45] macro expansion
      @ ~/.julia/juliaup/julia-1.8.0-rc1+0~x64/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [46] top-level scope
      @ ~/.julia/dev/ApproximateGPs.jl/test/SparseVariationalApproximationModule.jl:2
   [47] include(fname::String)
      @ Base.MainInclude ./client.jl:476
   [48] macro expansion
      @ ~/.julia/dev/ApproximateGPs.jl/test/runtests.jl:54 [inlined]
   [49] macro expansion
      @ ~/.julia/juliaup/julia-1.8.0-rc1+0~x64/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [50] top-level scope
      @ ~/.julia/dev/ApproximateGPs.jl/test/runtests.jl:54
   [51] include(fname::String)
      @ Base.MainInclude ./client.jl:476
   [52] top-level scope
      @ none:6
   [53] eval
      @ ./boot.jl:368 [inlined]
   [54] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:276

[devmotion]

Maybe the same as FluxML/Zygote.jl#1240 (comment)?

[devmotion]

Interestingly, this linked Zygote issue seems to present only on Julia nightly, so some change on the Julia master branch must be responsible for it.

[devmotion]

Tested on Julia 1.8-rc1.

Did you check if tests pass with 1.6 and 1.7?

[simsurace]

Not yet, but maybe you could just re-run the CI for the last commit on master.

[simsurace]

Julia 1.6 and 1.7 pass the tests with Zygote 0.6.41.

[devmotion]

So it really seems to be the upstream issue. Probably easiest to not use Julia 1.8 for now but of course that's not a good long term solution 😄

[simsurace]

Ah, is passing the tests for Julia 1.8 not required for PRs in Zygote to be merged? It seems that this would have been caught by having a must-pass CI run for 1.8.0-rc1.

[devmotion]

No, tests don't have to pass with unreleased versions/Julia master. I assume things would just be broken too often, that's at least what I observed in Turing and other packages that make use of AD, compiler tricks, etc.

[simsurace]

I think this makes perfect sense for the nightly build. But for the release candidate I think it would make sense to be testing already and making sure that new additions are already compatible to be fully ready once it is released, plus potentially allowing to discover more bugs in Julia before it is released. Otherwise as soon as it is released your current release of Zygote is going to be broken (as the CI for version '1' will now be pointing to 1.8).

[simsurace]

When I re-run the tests locally with the most recent Zygote version, the above error no longer occurs.