[lvaroqui]

CodSpeed::start_benchmark and end_benchmark now take a timestamp right after the start command and right after the benchmarked code returns, both outside of the measured section, and send them as RoundStart/RoundEnd markers before the stop command.

The runner times a benchmark when it receives the commands that start and stop it, a while before the benchmark starts and after it ends. On a local run of the divan telemetry example, the runner's sample of a benchmark lasted 128–310 µs, while the code of the benchmarks emitting telemetry took 2–43 µs. The rounds bound the benchmark closely, so the runner can measure what happens during it from them, e.g. place the telemetry a benchmark emits.

The markers are sent whatever the instrumentation status: instrument hooks drops them outside of the CodSpeed runner and under Valgrind, so simulation measurements are unaffected. They are only sent by the builds that run each benchmark once, under cfg(codspeed): the compat layers only reach this code there, but the native harness (codspeed_bench!) also runs in walltime builds, whose rounds are the walltime loop's.

In memory mode, the runner forwards the markers with the memory results, and CodSpeed does not accept round markers there yet. The backend support has to be deployed before this is released, or memory runs lose their results.

Refs COD-3862: the issue also tracks the benchmark telemetry built on top of this.

[codspeed]

Merging this PR will degrade performance by 4.93%

3 warnings, 1 note

⚠️ Unable to measure the execution of 59 benchmarks

These benchmarks measured no execution time, so their results are not comparable and count as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚠️ 17 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

ℹ️ 57 benchmarks have no flamegraph

• The flamegraph wasn't generated: most of the time was spent in runtime internals, which would leave nothing useful to analyze. (9 benchmarks)
• The flamegraph wasn't generated: no execution data was recorded while the benchmark ran. Make sure the compiler didn't optimize the benchmark out. (48 benchmarks)

See affected benchmarks

⚡ 11 improved benchmarks
❌ 21 regressed benchmarks
✅ 574 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Walltime	bench_array2[42]	35 ns	95 ns	-63.16%
❌	Walltime	hamiltonian_cycle[7]	1 µs	1.8 µs	-43.65%
❌	Walltime	div	3 ns	4 ns	-25%
❌	CPU Simulation	iterative[5]	234.1 ns	289.6 ns	-19.18%
❌	CPU Simulation	iterative[10]	245.4 ns	300.9 ns	-18.46%
❌	CPU Simulation	iterative[20]	267.9 ns	323.5 ns	-17.17%
❌	CPU Simulation	add_two_integers[(42, 13)]	288 ns	346 ns	-16.78%
❌	CPU Simulation	add_two_integers[(255, 255)]	288 ns	346 ns	-16.78%
❌	Walltime	rem	5 ns	6 ns	-16.67%
❌	CPU Simulation	iterative[30]	290.4 ns	346 ns	-16.06%
❌	CPU Simulation	add_two_integers[(1024, -1024)]	349.4 ns	407.7 ns	-14.31%
❌	CPU Simulation	add_two_integers[(65535, -65535)]	361.1 ns	419.5 ns	-13.91%
❌	Walltime	iter_with_setup	40 ns	45 ns	-11.11%
❌	Walltime	graph_coloring[4]	986 ns	1,091 ns	-9.62%
❌	CPU Simulation	a	568.5 ns	624.2 ns	-8.93%
❌	CPU Simulation	a	568.5 ns	624.2 ns	-8.93%
❌	CPU Simulation	iter_manual_simple	839.4 ns	894.9 ns	-6.21%
❌	CPU Simulation	process_items_with_counter	1.9 µs	2 µs	-5.42%
❌	Walltime	add_two_integers[(255, 255)]	20 ns	21 ns	-4.76%
❌	Walltime	zeroed_buffer[1]	47.9 µs	49.9 µs	-4.04%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing feat/time-benchmarks-with-round-markers (5b54240) with main (5caa8dc)}

[greptile-apps]

Confidence Score: 4/5

[Medium impact] The change reaches benchmark measurement across CodSpeed harnesses.

The timestamp change appears sound, but the added println! must satisfy the repository’s explicit output rule before merging.

Findings

1.  Added output breaks repository rule ▶

Fix with agent prompt

### Issue 1
crates/codspeed/build.rs:3
The new `println!` in `build.rs` violates the repository directive: “Never use `println!`/`eprintln!` in Rust; use tracing macros.” This requirement must be satisfied before merging. Use an allowed way to write the Cargo directive while preserving the stdout output Cargo needs.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds closely bounded round timestamps to benchmarks built with cfg(codspeed).

• Benchmark runs now mark the code with start and end timestamps.

The added println! violates the repository’s output rule.

Acknowledgment: lvaroqui states that memory results require backend support before release because the backend does not yet accept these round markers. This known release dependency is not reported again.

Diagram

sequenceDiagram
    participant Harness
    participant Runtime as CodSpeed
    participant Hooks as Instrument hooks
    Harness->>Runtime: start_benchmark
    Runtime->>Hooks: Start command
    Note over Runtime: Save round_start
    Note over Runtime: Start direct measurement
    Harness->>Harness: Run benchmark
    Harness->>Runtime: end_benchmark
    Note over Runtime: Stop direct measurement
    Note over Runtime: Capture round_end
    Runtime->>Hooks: Send round timestamps
    Runtime->>Hooks: Stop command

Loading

_{Reviews (1) · Last reviewed commit: "feat(codspeed): time each benchmark with..." · Reviewed by Greptile}

[not-matthias]

Looks good, just small comments about comments :)