Skip to content

Loading BlockTensorKit triggers severe gradient compilation slowdown for ordinary TensorMaps #86

Open

Description

@Yue-Zhengyuan

Loading BlockTensorKit makes the first gradient below fail to finish within 60 seconds.
Removing only the BlockTensorKit import, with the same project and input seed, avoids the slowdown.
The inputs contain no block tensors.

using LinearAlgebra, Random, TensorKit, BlockTensorKit, Zygote

BLAS.set_num_threads(1)
Random.seed!(123)

V = ComplexSpace(2)
x = (
    randn(ComplexF64, V ⊗ V ⊗ V ← V),
    randn(ComplexF64, V ← V ⊗ V),
)

function loss(x)
    a, b = adjoint.(x)
    return real(dot(a, a) + dot(b, b))
end

@time g = only(Zygote.gradient(loss, x))
@assert all(isapprox.(g, 2 .* x))

Tested on Julia 1.13.2, macOS arm64 / Apple M5, using AppleAccelerate and one Julia/BLAS thread.
Versions: TensorKit 0.17.2, BlockTensorKit 0.3.20, Zygote 0.7.13, ChainRules 1.73.0, ChainRulesCore 1.26.1.

Each measurement uses a fresh process and excludes package loading:

Case First gradient
Original example >60 s, interrupted
Without importing BlockTensorKit 2.83 s
Replace broadcast with a, b = x[1]', x[2]' 2.42 s
TensorKit experiment below 3.35 s
ChainRules experiment below 3.10 s

All completed cases produced the expected gradients.
These are individual first-call measurements, not benchmark averages.

Sampling shows compiler inference activity.
The broadcast pullback produces an abstractly typed cotangent vector, and inference visits block/sparse-block tensor paths despite the ordinary TensorMap inputs.
Loading BlockTensorKit is therefore a confirmed trigger, but I haven’t isolated a faulty method or established the root cause.

Two changes independently avoid the slowdown:

TensorKit: In generic VectorInterface.add, replace the chained mutation return with:

https://github.com/QuantumKitHub/TensorKit.jl/blob/a2add70e70663f6ebce06c52b003652a0331b18a/src/tensors/vectorinterface.jl#L82

- return add!(scale!(tdst, ty, β), tx, α)
+ scale!(tdst, ty, β)
+ add!(tdst, tx, α)
+ return tdst

ChainRules: In tuple unbroadcast, avoid inferring an unnecessary trailing-axis reduction for vectors:

https://github.com/JuliaDiff/ChainRules.jl/blob/5f308858e4e23db9b4ab525631402b08e3862f32/src/rulesets/Base/broadcast.jl#L338

- val = if N == length(dx)
+ val = if dx isa AbstractVector || N == length(dx)

Both are experimental candidates; I’m unsure whether either is the appropriate general fix.
The ChainRules change also seems worth discussing upstream.

Activity

  1. lkdvos commented on Oct 17, 2026

    @lkdvos
    Member

    Thanks for reporting! I'm not entirely sure what's going on, but it sounds like some invalidations must be the cause. I'm not at my desk this week and the next so I won't have time to inspect, but it would also be interesting to know if it is specifically the last update that caused this or if it was already the case before

  2. lkdvos commented on Oct 17, 2026

    @lkdvos
    Member

    The TensorKit change is interesting to evaluate as well, it sounds like inference works a lot better if we make mutating functions return the original argument, rather than depending on the inplace functions in the body returning the destination. I'm not sure if this would be a measurable improvement if we just do this in general, because it is a code pattern that comes up quite often

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions