You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
BMM quantization silently skipped when attention uses SDPA — no aten::bmm/aten::matmul nodes to detect #208
When a model's attention layers use scaled_dot_product_attention (SDPA) instead of eager attention, qmodel_prep silently fails to attach QBmm to any attention matmuls — even with nbits_bmm1/nbits_bmm2 correctly configured. SDPA is traced as a single fused kernel call rather than decomposed bmm/matmul ops, so BMM detection finds nothing. The only indication is an easy-to-miss INFO-level log line:
Found 0 torch.bmm and 0 torch.matmul
No warning or error is raised, so a user can end up with bmm or matmul left entirely unquantized in an otherwise-quantized model. Bug was only found while looking at the quantized model output.
For reproducibility, any HF transformers model whose attention defaults to SDPA (e.g. PatchTSTForPrediction on transformers>=4.53) run through qmodel_prep with BMM quantization enabled (nbits_bmm1/nbits_bmm2 set).
Expected behavior
torchscript tracer should have found 6 matmul operations. 0 was found as mentioned above.
Observed behavior
Found 0 torch.bmm and 0 torch.matmul
Additional context
A workaround is to set attn_implementation="eager" at model construction. A Warning can be issued in the interim, alerting users that if SDPA is used, no bmm and matmul ops may be found.
Hi! I'd like to take the interim step suggested here: log a warning in the TorchScript path of qmodel_prep when no bmm/matmul are found and SDPA is present in the graph (the Dynamo path already does this), pointing users to attn_implementation="eager", plus unit tests. I have a working fix and will open a PR shortly. I'm not planning to add SDPA support itself in this PR.
Following up on the interim warning, which is now merged in #214: I prototyped the full fix outside the repo. A replacement for F.scaled_dot_product_attention that does matmul → softmax → matmul, routing both matmuls through the existing QBmm modules, matches PyTorch's SDPA to ~1e-16 (float64) incl. masks/causal/GQA, and with the same QBmm instances it reproduces the eager+QBmm output exactly (0.0 diff, tiny BERT, linears left in fp32). Open design questions: (1) QBmm attachment during prep for SDPA models, keyed per attention module rather than by line number; (2) calibration behavior; (3) the speed cost of dropping the fused kernel (fine for calibration/simulated quant?). Since TorchScript is being retired, I'd target the Dynamo path. Would you want this, and is the approach acceptable?
Activity
Prer1n1 commentedon Nov 1, 2026
Hi! I'd like to take the interim step suggested here: log a warning in the TorchScript path of qmodel_prep when no bmm/matmul are found and SDPA is present in the graph (the Dynamo path already does this), pointing users to attn_implementation="eager", plus unit tests. I have a working fix and will open a PR shortly. I'm not planning to add SDPA support itself in this PR.
fix: warn when TorchScript tracer finds no bmm/matmul due to SDPA
Prer1n1 commentedon Nov 19, 2026
Following up on the interim warning, which is now merged in #214: I prototyped the full fix outside the repo. A replacement for F.scaled_dot_product_attention that does matmul → softmax → matmul, routing both matmuls through the existing QBmm modules, matches PyTorch's SDPA to ~1e-16 (float64) incl. masks/causal/GQA, and with the same QBmm instances it reproduces the eager+QBmm output exactly (0.0 diff, tiny BERT, linears left in fp32). Open design questions: (1) QBmm attachment during prep for SDPA models, keyed per attention module rather than by line number; (2) calibration behavior; (3) the speed cost of dropping the fused kernel (fine for calibration/simulated quant?). Since TorchScript is being retired, I'd target the Dynamo path. Would you want this, and is the approach acceptable?