Skip to content

BMM quantization silently skipped when attention uses SDPA — no aten::bmm/aten::matmul nodes to detect #208

Open

Description

@lorsonb

Description of the bug

When a model's attention layers use scaled_dot_product_attention (SDPA) instead of eager attention, qmodel_prep silently fails to attach QBmm to any attention matmuls — even with nbits_bmm1/nbits_bmm2 correctly configured. SDPA is traced as a single fused kernel call rather than decomposed bmm/matmul ops, so BMM detection finds nothing. The only indication is an easy-to-miss INFO-level log line:

Found 0 torch.bmm and 0 torch.matmul

No warning or error is raised, so a user can end up with bmm or matmul left entirely unquantized in an otherwise-quantized model. Bug was only found while looking at the quantized model output.

Platform

Python 3.12
fms-model-optimizer 0.8.5
transformers 5.12

Sample Code

For reproducibility, any HF transformers model whose attention defaults to SDPA (e.g. PatchTSTForPrediction on transformers>=4.53) run through qmodel_prep with BMM quantization enabled (nbits_bmm1/nbits_bmm2 set).

Expected behavior

torchscript tracer should have found 6 matmul operations. 0 was found as mentioned above.

Observed behavior

Found 0 torch.bmm and 0 torch.matmul

Additional context

A workaround is to set attn_implementation="eager" at model construction. A Warning can be issued in the interim, alerting users that if SDPA is used, no bmm and matmul ops may be found.

Activity

  1. Prer1n1 commented on Nov 1, 2026

    @Prer1n1
    Contributor

    Hi! I'd like to take the interim step suggested here: log a warning in the TorchScript path of qmodel_prep when no bmm/matmul are found and SDPA is present in the graph (the Dynamo path already does this), pointing users to attn_implementation="eager", plus unit tests. I have a working fix and will open a PR shortly. I'm not planning to add SDPA support itself in this PR.

  2. Prer1n1 commented on Nov 19, 2026

    @Prer1n1
    Contributor

    Following up on the interim warning, which is now merged in #214: I prototyped the full fix outside the repo. A replacement for F.scaled_dot_product_attention that does matmul → softmax → matmul, routing both matmuls through the existing QBmm modules, matches PyTorch's SDPA to ~1e-16 (float64) incl. masks/causal/GQA, and with the same QBmm instances it reproduces the eager+QBmm output exactly (0.0 diff, tiny BERT, linears left in fp32). Open design questions: (1) QBmm attachment during prep for SDPA models, keyed per attention module rather than by line number; (2) calibration behavior; (3) the speed cost of dropping the fused kernel (fine for calibration/simulated quant?). Since TorchScript is being retired, I'd target the Dynamo path. Would you want this, and is the approach acceptable?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions