From b8c01ccd520545caa2ba5c0a179bbf4a043f1451 Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Wed, 19 Aug 2026 01:00:30 +0000 Subject: [PATCH 1/2] add reporting guide: bug reports and performance claims Co-Authored-By: Claude Fable 5 --- .ai/AGENTS.md | 1 + .ai/reporting.md | 59 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+) create mode 100644 .ai/reporting.md diff --git a/.ai/AGENTS.md b/.ai/AGENTS.md index ae611feb4bb6..cfec97b60d75 100644 --- a/.ai/AGENTS.md +++ b/.ai/AGENTS.md @@ -31,6 +31,7 @@ Strive to write code as simple and explicit as possible. - **Pipelines** — see [pipelines.md](pipelines.md) for pipeline conventions, patterns, and gotchas. - **Modular pipelines** — see [modular.md](modular.md) for modular pipeline conventions, patterns, and gotchas. - **Tests** — see [testing.md](testing.md) for test conventions: required test layers, tester mixins, and dummy-component rules. +- **Reporting** — see [reporting.md](reporting.md) for how to write bug reports (what a reproduction is) and performance claims (end-to-end numbers first). ## Skills diff --git a/.ai/reporting.md b/.ai/reporting.md new file mode 100644 index 000000000000..efb81d2a72d2 --- /dev/null +++ b/.ai/reporting.md @@ -0,0 +1,59 @@ +# Reporting bugs and performance claims + +For issues and PR descriptions on this repo, from humans and agents alike. + +## Bug reports + +A reproduction is **what you were doing when it broke, with everything unnecessary removed** — start from the real failing situation and delete, don't build a clean synthetic case from scratch. The test: can someone paste it into a terminal and see the same failure, without first accepting your theory of the cause? A script that demonstrates your theory is not a reproduction — it can pass while the real bug is still there. + +- Keep the real model, settings, and dtype. A repro that downloads weights and takes two minutes is worth more than a fast synthetic one. +- If you truly can't produce one (gated model, 8 GPUs), say so at the top and give the closest thing you have. Don't silently substitute something smaller. + +✓ A reproduction — the failing call itself, trimmed (from [#14518](https://github.com/huggingface/diffusers/issues/14518): Krea-2 OOMs a 16GB GPU under 4-bit quantization): + +```python +import torch +from diffusers import Krea2Pipeline + +pipe = Krea2Pipeline.from_pretrained(...) +pipe.to("cuda") + +image = pipe(...).images[0] + +print(f"peak allocated: {torch.cuda.max_memory_allocated() / 2**30:.2f} GiB") # 19.94 GiB — OOM on 16GB +``` + +✗ Not a reproduction — a synthetic benchmark built to demonstrate the suspected cause (paraphrased from an earlier draft of the same report): + +```python +# "enable_gqa + attn_mask falls back to the math backend and materializes an S x S score matrix" +q = torch.randn(1, 24, 8100, 128, device="cuda", dtype=torch.bfloat16) +k = v = torch.randn(1, 6, 8100, 128, device="cuda", dtype=torch.bfloat16) +mask = torch.ones(1, 1, 8100, 8100, dtype=torch.bool, device="cuda") +F.scaled_dot_product_attention(q, k, v, attn_mask=mask, enable_gqa=True) +print(torch.cuda.max_memory_allocated()) +``` + +The difference: a maintainer understands the first script at a glance — it's an ordinary pipeline call — while the second is not, and it only demonstrates the reporter's theory: it can go green while the real pipeline still dies on a 24GB card. + +For structure, follow the [bug report template](../.github/ISSUE_TEMPLATE/bug-report.yml). + +## Performance claims + +Report **the end-to-end number only**: wall clock for the full pipeline call, before vs. after, on the same hardware, dtype, and seed. That's the number we decide with — if we want op-level detail, we'll ask. + +Attach the script you measured with, please keep it as simple as possible (see `benchmarks/benchmarking_utils.py`): + +```python +import torch.utils.benchmark as benchmark + +def benchmark_fn(f, *args, **kwargs): + t0 = benchmark.Timer(stmt="f(*args, **kwargs)", globals={"args": args, "kwargs": kwargs, "f": f}, num_threads=1) + return f"{t0.blocked_autorange().mean:.3f}s" + +pipe(**call_kwargs) # warmup +print(benchmark_fn(pipe, **call_kwargs)) +print(f"peak memory: {torch.cuda.max_memory_allocated() / 1024**3:.2f}GB") +``` + +State the setup (GPU, dtype, torch/diffusers versions, attention backend) and the exact call (model id, resolution/duration, steps, batch size). One-shot timings are noise. From aabca05ab804a04dac521483cc9fb369711b73d9 Mon Sep 17 00:00:00 2001 From: YiYi Xu Date: Wed, 26 Aug 2026 16:52:32 -1000 Subject: [PATCH 2/2] Apply suggestions from code review Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --- .ai/reporting.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/.ai/reporting.md b/.ai/reporting.md index efb81d2a72d2..e26471d89a23 100644 --- a/.ai/reporting.md +++ b/.ai/reporting.md @@ -1,12 +1,12 @@ # Reporting bugs and performance claims -For issues and PR descriptions on this repo, from humans and agents alike. +Use this guide when writing bug reports or performance claims in issues and pull requests. ## Bug reports -A reproduction is **what you were doing when it broke, with everything unnecessary removed** — start from the real failing situation and delete, don't build a clean synthetic case from scratch. The test: can someone paste it into a terminal and see the same failure, without first accepting your theory of the cause? A script that demonstrates your theory is not a reproduction — it can pass while the real bug is still there. +A reproduction is the smallest version of the real workflow that still fails. Start with what you were doing when the failure occurred, then remove anything unrelated. Someone should be able to run it and see the same failure without first accepting your theory about its cause. A script that tests your theory is not a reproduction. It may pass even when the real workflow still fails. -- Keep the real model, settings, and dtype. A repro that downloads weights and takes two minutes is worth more than a fast synthetic one. +- Keep the real model, settings, and dtype. A slower repro is better than a faster one that changes the behavior. - If you truly can't produce one (gated model, 8 GPUs), say so at the top and give the closest thing you have. Don't silently substitute something smaller. ✓ A reproduction — the failing call itself, trimmed (from [#14518](https://github.com/huggingface/diffusers/issues/14518): Krea-2 OOMs a 16GB GPU under 4-bit quantization): @@ -34,15 +34,15 @@ F.scaled_dot_product_attention(q, k, v, attn_mask=mask, enable_gqa=True) print(torch.cuda.max_memory_allocated()) ``` -The difference: a maintainer understands the first script at a glance — it's an ordinary pipeline call — while the second is not, and it only demonstrates the reporter's theory: it can go green while the real pipeline still dies on a 24GB card. +The first script exposes the real pipeline call. The second only tests the proposed mechanism, so it may pass even when the pipeline still fails. Keep the hypothesis separate from the reproduction. For structure, follow the [bug report template](../.github/ISSUE_TEMPLATE/bug-report.yml). ## Performance claims -Report **the end-to-end number only**: wall clock for the full pipeline call, before vs. after, on the same hardware, dtype, and seed. That's the number we decide with — if we want op-level detail, we'll ask. +Lead with the end-to-end result: measure the full pipeline call before and after the change on the same hardware, dtype, and seed. Only include lower-level measurements when requested. -Attach the script you measured with, please keep it as simple as possible (see `benchmarks/benchmarking_utils.py`): +Attach the measurement script and keep it as simple as possible. See `benchmarks/benchmarking_utils.py` for the repository's benchmarking helper. ```python import torch.utils.benchmark as benchmark @@ -56,4 +56,4 @@ print(benchmark_fn(pipe, **call_kwargs)) print(f"peak memory: {torch.cuda.max_memory_allocated() / 1024**3:.2f}GB") ``` -State the setup (GPU, dtype, torch/diffusers versions, attention backend) and the exact call (model id, resolution/duration, steps, batch size). One-shot timings are noise. +State the setup (GPU, dtype, torch/diffusers versions, attention backend) and the exact call (model id, resolution/duration, steps, batch size). One-shot timings are noisy.