Skip to content

[CMake 6/7] Initialize CVODES adjoint quadrature derivative buffers - #3508

Merged
WardBrian merged 2 commits into
developfrom
fix/cvodes-adjoint-quadrature-output
Nov 20, 2026
Merged

WardBrian merged 2 commits into
developfrom
fix/cvodes-adjoint-quadrature-output

Conversation

@SteveBronder

@SteveBronder SteveBronder commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The adjoint cosine ODE tests fail on Windows x64 with incorrect gradients and on Windows ARM64 with CVodeB error -3 in the CMake CI run. CVODES reuses its quadrature derivative output buffer, but accumulate_adjoints adds to that buffer with +=, retaining stale or uninitialized values between callbacks.

Clear the output before accumulating derivatives. Add an analytic value/gradient regression covering scalar and vector parameters, repeated solves, and repeated reverse passes. This is a solver correctness fix reviewed separately from the CMake infrastructure; existing tolerances stay unchanged.

Tests

  • Observed failures before the fix in the existing adjoint tests and the new regression when allocator poisoning fills reused memory with nonzero bytes (MALLOC_PERTURB_).
  • After the fix, all 56 cosine ODE cases pass ten shuffled repetitions with allocator poisoning; the 12 adjoint/regression cases pass ten repetitions with a second poison value.
  • ARM64 Linux under QEMU: the regression fails before the fix and passes three repetitions after it.
  • Windows x64 and ARM64 LLVM-Mingw cross-compilation passes with -Wall -Werror; native Windows runtime validation remains for CI.
  • Dependency checks, cpplint, and git diff --check pass.

Side Effects

Adjoint quadrature derivatives no longer depend on previous output-buffer contents. Public APIs and solver tolerances are unchanged.

Release notes

Fix incorrect adjoint ODE parameter gradients and intermittent CVODES backward-integration failures caused by stale quadrature output.

Checklist

  • Copyright holder: Simons Foundation
  • New value and gradient regression added
  • Focused validation passes as described above
  • Complete CI matrix passes
  • C++ idioms and Doxygen documentation have completed review

Related PRs

Native stack: https://github.com/stan-dev/math/stack/3509

@jachymb

jachymb commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Thanks for tracking this down. Some observations, in case they help:

It also hits the current CI. In #3446, which does not touch ODE code, rev tests (windows-11-arm) failed StanOde/cos_arg_test/4.tol_grad with a.adj() = -0.43164793 instead of -0.43164769. The same job passed on develop and, a minute earlier, on another PR with the same base commit.

On develop the gradient can be silently wrong by any amount. I ran a one-parameter example (dy/dt = -a t y, all tolerances 1e-10) under glibc's MALLOC_PERTURB_ (WSL, g++ 14.2, glibc 2.41, -O3):

new memory reads as relative error of dy/da on develop
0 8.1e-10
1.6e11 1.5e-6
7.5e20 0.11
1.7e40 5e8
1.4e306 throws (CVodeGetQuadB, error flag -25)

With the N_VConst(0.0, qBdot) line from this PR the result is the same for every fill (relative error 7.8e-10), and cos_ode_typed_test passes 55 of 55 under all ten fill settings I tried.

The callback has been like this since v4.1.0.

Example program

Saved as test/unit/math/rev/functor/cvodes_adjoint_quad_buffer_test.cpp and run with MALLOC_PERTURB_ = 255, 189, 187, 183, 128 for the rows above.

#include <stan/math/rev.hpp>
#include <gtest/gtest.h>
#include <cmath>
#include <cstdio>
#include <vector>

// dy/dt = -a t y  =>  y(T) = y0 exp(-a (T^2 - t0^2) / 2)
struct decay_rhs {
  template <typename T_t, typename T_y, typename T_a>
  Eigen::Matrix<stan::return_type_t<T_y, T_a>, -1, 1> operator()(
      const T_t& t, const T_y& y, std::ostream* msgs, const T_a& a) const {
    return -a * y * t;
  }
};

TEST(cvodesAdjoint, quadrature_buffer) {
  const double t0 = 0.5, T = 1.25, tol = 1e-10;
  const Eigen::VectorXd y0 = Eigen::VectorXd::Constant(1, 0.75);
  const Eigen::VectorXd atol = Eigen::VectorXd::Constant(1, tol);
  stan::math::var a = 0.2;
  auto y = stan::math::ode_adjoint_tol_ctl(
      decay_rhs(), y0, t0, std::vector<double>{T}, tol, atol, tol, atol, tol,
      tol, 1000000, 150, CV_HERMITE, CV_BDF, CV_ADAMS, nullptr, a);
  y[0](0).grad();
  const double c = -0.5 * (T * T - t0 * t0);
  const double exact = c * y0(0) * std::exp(c * a.val());
  std::printf("dy/da = %.17g, relative error %.2g\n", a.adj(),
              std::abs(a.adj() / exact - 1));
}

@WardBrian
WardBrian force-pushed the fix/cvodes-adjoint-quadrature-output branch from 77455ad to 01c1ec9 Compare November 13, 2026 12:42
@WardBrian
WardBrian force-pushed the fix/cvodes-adjoint-quadrature-output branch from 01c1ec9 to 87c42dd Compare November 17, 2026 14:49
Base automatically changed from cmake-split/05-unit-tests to develop November 17, 2026 22:13
@WardBrian
WardBrian force-pushed the fix/cvodes-adjoint-quadrature-output branch from 87c42dd to 3caa4e0 Compare November 17, 2026 22:13
Clear the quadrature derivative output before accumulating adjoints.
Cover analytic values and gradients across repeated solves and reverse passes.
@SteveBronder
SteveBronder force-pushed the fix/cvodes-adjoint-quadrature-output branch from 3caa4e0 to 9bbd900 Compare November 19, 2026 17:02

@WardBrian WardBrian left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record, to get the test to actually fail without the patch I needed to run

for i in {1..100}; do MALLOC_PERTURB_=12 ./runTests.py -j12 test/unit/math/rev/functor/cos_ode_typed_test.cpp ; done

@WardBrian
WardBrian enabled stacked auto-merge November 19, 2026 19:09
@stan-buildbot

Copy link
Copy Markdown
Contributor
Name Old Result New Result Ratio Performance change( 1 - new / old )
stat_comp_benchmarks/benchmarks/gp_regr/gp_regr.stan 0.24 0.25 0.99 -1.23% slower
stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan 0.06 0.06 1.01 1.29% faster
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix/low_dim_gauss_mix.stan 6.67 6.62 1.01 0.69% faster
stat_comp_benchmarks/benchmarks/low_dim_corr_gauss/low_dim_corr_gauss.stan 0.02 0.02 0.98 -2.21% slower
stat_comp_benchmarks/benchmarks/irt_2pl/irt_2pl.stan 9.24 9.29 0.99 -0.62% slower
stat_comp_benchmarks/benchmarks/gp_pois_regr/gp_pois_regr.stan 4.89 4.66 1.05 4.66% faster
stat_comp_benchmarks/benchmarks/sir/sir.stan 226.43 223.99 1.01 1.08% faster
stat_comp_benchmarks/benchmarks/garch/garch.stan 1.0 1.0 1.0 0.42% faster
stat_comp_benchmarks/benchmarks/arma/arma.stan 0.69 0.69 1.0 0.11% faster
stat_comp_benchmarks/benchmarks/pkpd/one_comp_mm_elim_abs.stan 42.27 43.23 0.98 -2.28% slower
stat_comp_benchmarks/benchmarks/pkpd/sim_one_comp_mm_elim_abs.stan 0.58 0.6 0.97 -3.05% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 21.76 21.89 0.99 -0.57% slower
stat_comp_benchmarks/benchmarks/eight_schools/eight_schools.stan 0.11 0.11 1.01 1.32% faster
stat_comp_benchmarks/benchmarks/arK/arK.stan 3.4 3.4 1.0 0.08% faster
performance.compilation 355.1 397.82 0.89 -12.03% slower
Mean result: 0.9929540621613376

Jenkins Console Log
Jenkins Build Stages
Commit hash: f89dcc16d4f764402a709a89cd2371ea988bf4c4

Machine information
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

CPU:

Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Byte Order:                              Little Endian
Address sizes:                           43 bits physical, 48 bits virtual
CPU(s):                                  256
On-line CPU(s) list:                     0-255
Thread(s) per core:                      2
Core(s) per socket:                      64
Socket(s):                               2
NUMA node(s):                            2
Vendor ID:                               AuthenticAMD
CPU family:                              23
Model:                                   49
Model name:                              AMD EPYC 7742 64-Core Processor
Stepping:                                0
Frequency boost:                         enabled
CPU MHz:                                 1496.790
CPU max MHz:                             3416.0681
CPU min MHz:                             1500.0000
BogoMIPS:                                4491.80
Virtualization:                          AMD-V
L1d cache:                               4 MiB
L1i cache:                               4 MiB
L2 cache:                                64 MiB
L3 cache:                                512 MiB
NUMA node0 CPU(s):                       0-63,128-191
NUMA node1 CPU(s):                       64-127,192-255
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es

G++:

g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:

clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

@WardBrian
WardBrian merged commit 3d16e8f into develop Nov 20, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants